2024-03-25T18:22:07.516 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:22:07.516 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --verbose -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/orig.config.yaml 2024-03-25T18:22:07.557 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:22:07.642 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621326' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQCzV6lpXKIBd4N/cJBTgzahLvMCwQAhXWL0LgcI3DO1snJHDmoyF4ex80DRwIxvCTQ+hZ070f+4Ry0HY0P/PM= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCknM59JyyEIUSXB1BqSUvecxOxnQEhCaT3/GXMIpAOQkfek8WYgZoMpKoTlmZYX5bmosbyObRrIa5xHthxUpxI= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:22:07.642 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:22:07.642 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:22:07.643 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:22:07.644 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:22:07.665 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:22:07.665 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:22:07.665 INFO:teuthology.packaging:ref: None 2024-03-25T18:22:07.665 INFO:teuthology.packaging:tag: None 2024-03-25T18:22:07.665 INFO:teuthology.packaging:branch: main 2024-03-25T18:22:07.665 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:22:07.666 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:22:08.023 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:22:08.025 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:22:08.031 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:22:08.031 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:22:08.049 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:22:08.064 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:22:08.071 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:22:08.096 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:12.047698', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQCzV6lpXKIBd4N/cJBTgzahLvMCwQAhXWL0LgcI3DO1snJHDmoyF4ex80DRwIxvCTQ+hZ070f+4Ry0HY0P/PM='} 2024-03-25T18:22:08.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:12.049502', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCknM59JyyEIUSXB1BqSUvecxOxnQEhCaT3/GXMIpAOQkfek8WYgZoMpKoTlmZYX5bmosbyObRrIa5xHthxUpxI='} 2024-03-25T18:22:08.120 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:22:08.128 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:22:08.128 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:22:08.128 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:22:08.201 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f08c493e0d0>, signals=[15]) 2024-03-25T18:22:08.201 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:22:08.208 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:22:08.208 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T18:22:08.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:08.298 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:22:08.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:08.383 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:22:08.391 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-03-25T18:22:08.408 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-03-25T18:22:08.408 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-03-25T18:22:08.464 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-03-25T18:22:08.464 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:22:08.465 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:22:08.466 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-03-25T18:22:08.484 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-03-25T18:22:08.502 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-03-25T18:22:08.502 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-03-25T18:22:08.557 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:22:08.558 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:22:08.559 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:22:08.559 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-03-25T18:22:08.584 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:22:08.595 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:22:08.603 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:22:08.603 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:22:08.605 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:22:08.622 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:22:08.629 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:22:08.629 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:22:08.662 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:22:08.765 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:22:08.823 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:22:08.839 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:22:08.839 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:22:08.839 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:22:08.839 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:22:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:08.857 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-03-25T18:22:08.857 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-25T18:22:08.913 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:22:08.913 INFO:teuthology.task.kernel:Running kernel on smithi193: 5.14.0-361.el9.x86_64 2024-03-25T18:22:08.913 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2024-03-25T18:22:08.914 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:22:08.915 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-361.el9.x86_64 2024-03-25T18:22:08.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-03-25T18:22:11.205 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T18:22:11.496 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-03-25T18:22:12.847 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T18:22:13.104 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-25T18:22:17.084 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:22:17.317 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:22:22.246 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-03-25T18:22:22.553 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-03-25T18:22:29.256 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:22:29.588 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:22:30.694 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-25T18:22:31.294 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T18:22:32.815 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T18:22:32.920 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-03-25T18:22:37.703 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-25T18:22:37.795 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-25T18:22:39.826 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:22:39.827 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:22:39.915 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:22:39.922 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:22:39.922 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-03-25T18:22:39.923 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:22:39.924 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 76 M 2024-03-25T18:22:39.924 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 123 M 2024-03-25T18:22:39.924 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:22:39.927 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:22:39.927 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:22:40.020 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout:Install 4 Packages 2024-03-25T18:22:40.032 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:22:40.033 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 76 M 2024-03-25T18:22:40.033 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 123 M 2024-03-25T18:22:40.033 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:22:40.182 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:40.182 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:40.183 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:40.247 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:40.249 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:40.249 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:40.354 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.354 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.356 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.400 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:40.409 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:40.417 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.426 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.450 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:40.490 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.491 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.523 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.541 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.665 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.683 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.706 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.720 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.744 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.751 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:40.755 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:40.773 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.779 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:40.807 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:40.816 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.849 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.850 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.884 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.903 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:40.939 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:40.966 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:40.974 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.025 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:41.077 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:41.086 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:41.090 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.109 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.113 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.125 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.157 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:41.167 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.168 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.177 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:41.189 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:41.282 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.293 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:41.296 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.303 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:41.316 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:41.354 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:41.404 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 55 kB/s | 66 kB 00:01 2024-03-25T18:22:41.438 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:41.459 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:41.503 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:41.552 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:41.585 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:41.619 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:41.619 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:41.653 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:41.653 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:41.671 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:41.671 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:41.687 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:41.867 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:41.912 INFO:teuthology.orchestra.run.smithi193.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 38 kB/s | 66 kB 00:01 2024-03-25T18:22:42.020 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:42.055 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:42.073 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:42.288 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 16 MB/s | 33 MB 00:02 2024-03-25T18:22:42.292 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:42.301 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:42.311 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:42.346 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:42.480 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:42.521 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:42.535 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:42.574 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:42.609 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:42.650 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:42.846 INFO:teuthology.orchestra.run.smithi193.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 5.5 MB/s | 15 MB 00:02 2024-03-25T18:22:42.863 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 5.6 MB/s | 15 MB 00:02 2024-03-25T18:22:42.904 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:43.230 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:43.387 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:43.705 INFO:teuthology.orchestra.run.smithi193.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 9.4 MB/s | 33 MB 00:03 2024-03-25T18:22:43.707 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:43.762 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:43.980 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 11 MB/s | 27 MB 00:02 2024-03-25T18:22:43.985 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:43.985 INFO:teuthology.orchestra.run.smithi071.stdout:Total 19 MB/s | 76 MB 00:04 2024-03-25T18:22:44.649 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:22:44.780 INFO:teuthology.orchestra.run.smithi193.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 9.6 MB/s | 27 MB 00:02 2024-03-25T18:22:44.786 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:44.786 INFO:teuthology.orchestra.run.smithi193.stdout:Total 16 MB/s | 76 MB 00:04 2024-03-25T18:22:45.043 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:22:45.043 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:22:45.453 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:22:45.799 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:22:45.799 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:22:46.068 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:22:46.068 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:22:46.835 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:22:46.836 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:22:48.649 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:22:49.316 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:22:49.331 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:49.415 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:50.021 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:52.177 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:52.322 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:52.746 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:52.882 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:57.016 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:57.113 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:57.650 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:57.682 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:01.822 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:02.411 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:31.020 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:31.020 INFO:teuthology.orchestra.run.smithi071.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:31.021 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:23:31.065 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:31.396 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:31.397 INFO:teuthology.orchestra.run.smithi193.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:31.397 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:23:31.441 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:31.446 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:31.446 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:31.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:31.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:31.838 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:31.840 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:31.840 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:23:31.840 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:23:31.840 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:31.840 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:31.840 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:31.841 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:31.841 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:23:31.841 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:23:32.176 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:32.263 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:32.263 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:23:32.264 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:23:32.264 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:32.264 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:32.264 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:32.264 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:32.264 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:23:32.264 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:23:32.278 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:32.279 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:32.279 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:32.279 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:32.279 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:32.279 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:23:32.279 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:32.279 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:32.359 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:32.359 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:32.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-03-25T18:23:32.394 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-03-25T18:23:32.394 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-03-25T18:23:32.394 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2024-03-25T18:23:32.394 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 68.el9 2024-03-25T18:23:32.394 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-03-25T18:23:32.394 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:32.394 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2024-03-25T18:23:32.394 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8264254 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:23:32.395 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:32.396 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:32.396 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:32.396 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-03-25T18:23:32.396 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:23:32.396 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:32.397 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:32.397 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:32.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:32.588 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:32.724 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:32.725 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:32.725 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:32.725 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:32.725 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:32.726 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:23:32.726 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:32.726 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:32.804 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:32.804 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:32.804 DEBUG:teuthology.orchestra.run.smithi193:> sudo rpm -qi grub2-tools 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:Name : grub2-tools 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:Epoch : 1 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:Version : 2.06 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:Release : 68.el9 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: x86_64 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:Group : Unspecified 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:Size : 8264254 2024-03-25T18:23:32.838 INFO:teuthology.orchestra.run.smithi193.stdout:License : GPLv3+ 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:Packager : builder@centos.org 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:Vendor : CentOS 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:Description : 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:32.839 INFO:teuthology.orchestra.run.smithi193.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:32.840 INFO:teuthology.orchestra.run.smithi193.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:32.840 INFO:teuthology.orchestra.run.smithi193.stdout:hardware devices. 2024-03-25T18:23:32.840 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:23:32.840 INFO:teuthology.orchestra.run.smithi193.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:32.840 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:32.841 INFO:teuthology.task.kernel:Updating grub on smithi193 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:32.841 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:32.987 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-03-25T18:23:33.478 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2024-03-25T18:23:34.274 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:34.312 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-03-25T18:23:34.314 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-25T18:23:34.329 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.MiwRr1fdAb 2024-03-25T18:23:34.329 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MiwRr1fdAb 2024-03-25T18:23:34.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.MiwRr1fdAb 2024-03-25T18:23:34.536 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.MiwRr1fdAb is 6KB 2024-03-25T18:23:34.585 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.MiwRr1fdAb 2024-03-25T18:23:34.599 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:34.671 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:34.671 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:34.672 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:34.672 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:34.673 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:34.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-03-25T18:23:34.928 INFO:teuthology.orchestra.run.smithi193.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:34.962 INFO:teuthology.orchestra.run.smithi193.stderr:done 2024-03-25T18:23:34.964 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-03-25T18:23:34.979 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.C9rWVzToFG 2024-03-25T18:23:34.979 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.C9rWVzToFG 2024-03-25T18:23:35.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.C9rWVzToFG 2024-03-25T18:23:35.198 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.C9rWVzToFG is 6KB 2024-03-25T18:23:35.246 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.C9rWVzToFG 2024-03-25T18:23:35.261 DEBUG:teuthology.orchestra.run.smithi193:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:35.325 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:35.325 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:35.325 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:35.325 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:35.326 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:35.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2024-03-25T18:24:04.858 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:24:04.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-25T18:24:04.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:05.456 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:24:05.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T18:24:05.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:23.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-25T18:24:23.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2024-03-25T18:24:32.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-25T18:24:32.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:32.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T18:24:32.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:35.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-25T18:24:47.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-25T18:24:47.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:19.202 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-25T18:25:19.647 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-25T18:25:19.647 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:25:19.648 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-25T18:25:19.661 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:25:19.661 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:25:19.661 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:25:19.662 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:25:20.663 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:25:20.663 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:25:20.749 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-03-25T18:25:20.784 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:32.843 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:25:44.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T18:25:44.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:45.155 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-25T18:25:45.530 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T18:25:45.530 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:25:45.530 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-03-25T18:25:45.545 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:25:45.545 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:25:45.546 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:25:45.546 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:25:46.546 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:25:46.546 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:25:46.631 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2024-03-25T18:25:46.663 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:46.663 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:25:46.673 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:25:46.674 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:25:46.676 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:25:46.693 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:25:46.700 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:25:46.709 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:25:46.710 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:25:46.735 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:25:46.801 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:25:46.807 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:25:46.808 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:25:46.810 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:25:46.847 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:46.849 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:46.861 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:46.863 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:46.865 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:25:46.872 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:25:46.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:25:46.908 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:25:46.937 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:25:46.946 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:25:46.946 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:25:46.978 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:25:46.994 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:47.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:47.134 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:47.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:47.260 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T18:25:47.260 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:25:47.328 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:47.374 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:47.429 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:47.513 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:47.556 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:25:47.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:25:47.623 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-03-25T18:25:47.625 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-03-25T18:25:47.675 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:25:47.708 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:25:48.175 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:25:48.183 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:25:48.183 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:25:48.192 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:25:48.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-03-25T18:25:48.243 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-03-25T18:25:48.245 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-03-25T18:25:48.316 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-03-25T18:25:48.318 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:25:48.318 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-03-25T18:25:48.345 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-03-25T18:25:48.345 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-03-25T18:25:48.368 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-03-25T18:25:48.368 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:25:48.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:25:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:25:48.413 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:25:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:25:48.442 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:25:48.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:25:48.497 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:25:48.528 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:25:48.540 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:25:48.557 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:25:48.559 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:30:07.991 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-03-25T18:30:07.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-25T18:30:07.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:30:08.085 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-25T18:30:08.170 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-25T18:30:08.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T18:30:08.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:30:08.258 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-25T18:30:08.346 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T18:30:08.346 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:30:08.357 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:30:08.358 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:30:08.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:30:08.361 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:30:08.361 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:30:08.390 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:30:08.407 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:30:08.417 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:30:08.435 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:30:08.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-03-25T18:30:08.449 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-03-25T18:30:08.464 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-03-25T18:30:08.466 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:30:08.477 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-03-25T18:30:08.483 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:30:08.494 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:30:08.510 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:30:08.534 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:30:08.536 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:30:08.536 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-25T18:30:08.536 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:08.537 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:08.537 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:08.537 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:08.558 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:30:08.561 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:30:08.561 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-25T18:30:08.561 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:08.561 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:08.561 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:08.561 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:08.562 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:30:08.571 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:30:08.571 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:30:08.571 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:30:08.571 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:30:08.577 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:30:08.577 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:30:08.577 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:30:08.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:30:08.580 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:30:08.580 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:30:08.729 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:30:08.729 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:30:08.843 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:30:08.843 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:30:09.171 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:30:09.171 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T18:30:09.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:30:09.200 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:30:09.200 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:30:09.269 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:30:09.269 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:30:09.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:30:09.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:30:09.300 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:30:09.300 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:30:09.352 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:30:09.370 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:30:09.419 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-03-25T18:30:09.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-03-25T18:30:09.452 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:30:09.519 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-03-25T18:30:09.520 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-03-25T18:30:09.684 INFO:teuthology.orchestra.run.smithi071.stdout:74 files removed 2024-03-25T18:30:09.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2024-03-25T18:30:09.767 INFO:teuthology.orchestra.run.smithi193.stdout:74 files removed 2024-03-25T18:30:09.795 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw 2024-03-25T18:30:10.919 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 99 kB/s | 82 kB 00:00 2024-03-25T18:30:11.076 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 77 kB/s | 82 kB 00:01 2024-03-25T18:30:11.945 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 10 kB/s | 10 kB 00:00 2024-03-25T18:30:11.960 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:30:12.713 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-25T18:30:12.910 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-03-25T18:30:13.356 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T18:30:13.612 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T18:30:14.848 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-25T18:30:15.121 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-25T18:30:18.945 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:30:21.424 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 1.3 MB/s | 8.0 MB 00:05 2024-03-25T18:30:24.388 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-25T18:30:26.635 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-03-25T18:30:31.732 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-25T18:30:33.121 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-03-25T18:30:33.428 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.1 MB 00:03 2024-03-25T18:30:34.531 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T18:30:35.325 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-25T18:30:36.277 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-25T18:30:39.447 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-25T18:30:41.160 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-25T18:30:41.903 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:30:41.909 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:30:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:30:41.909 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:30:41.909 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:30:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:30:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout:Install 30 Packages 2024-03-25T18:30:41.912 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 2 Packages 2024-03-25T18:30:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:30:41.913 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 58 M 2024-03-25T18:30:41.913 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:30:43.662 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:30:43.676 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:30:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:30:43.676 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:30:43.676 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:30:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:30:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:30:43.679 INFO:teuthology.orchestra.run.smithi071.stdout:Install 30 Packages 2024-03-25T18:30:43.680 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2024-03-25T18:30:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:30:43.680 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 58 M 2024-03-25T18:30:43.680 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:30:57.733 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 396 kB/s | 5.3 MB 00:13 2024-03-25T18:30:57.900 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 151 kB/s | 25 kB 00:00 2024-03-25T18:30:59.130 INFO:teuthology.orchestra.run.smithi193.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 323 kB/s | 5.3 MB 00:16 2024-03-25T18:30:59.247 INFO:teuthology.orchestra.run.smithi193.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T18:31:00.100 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 337 kB/s | 742 kB 00:02 2024-03-25T18:31:01.272 INFO:teuthology.orchestra.run.smithi193.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 366 kB/s | 742 kB 00:02 2024-03-25T18:31:01.933 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 272 kB/s | 498 kB 00:01 2024-03-25T18:31:02.322 INFO:teuthology.orchestra.run.smithi193.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 475 kB/s | 498 kB 00:01 2024-03-25T18:31:05.664 INFO:teuthology.orchestra.run.smithi193.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 425 kB/s | 9.7 MB 00:23 2024-03-25T18:31:05.830 INFO:teuthology.orchestra.run.smithi193.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 271 kB/s | 45 kB 00:00 2024-03-25T18:31:06.155 INFO:teuthology.orchestra.run.smithi193.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 404 kB/s | 131 kB 00:00 2024-03-25T18:31:06.564 INFO:teuthology.orchestra.run.smithi193.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 396 kB/s | 162 kB 00:00 2024-03-25T18:31:07.008 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 9.7 MB 00:23 2024-03-25T18:31:07.216 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 217 kB/s | 45 kB 00:00 2024-03-25T18:31:07.322 INFO:teuthology.orchestra.run.smithi193.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 423 kB/s | 320 kB 00:00 2024-03-25T18:31:07.642 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 309 kB/s | 131 kB 00:00 2024-03-25T18:31:08.042 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 405 kB/s | 162 kB 00:00 2024-03-25T18:31:08.348 INFO:teuthology.orchestra.run.smithi193.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 290 kB/s | 298 kB 00:01 2024-03-25T18:31:08.756 INFO:teuthology.orchestra.run.smithi193.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 243 kB/s | 99 kB 00:00 2024-03-25T18:31:08.923 INFO:teuthology.orchestra.run.smithi193.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 636 kB/s | 106 kB 00:00 2024-03-25T18:31:08.973 INFO:teuthology.orchestra.run.smithi193.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:31:09.007 INFO:teuthology.orchestra.run.smithi193.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:31:09.074 INFO:teuthology.orchestra.run.smithi193.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T18:31:09.124 INFO:teuthology.orchestra.run.smithi193.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-25T18:31:09.158 INFO:teuthology.orchestra.run.smithi193.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:31:09.224 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:09.352 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:09.392 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 237 kB/s | 320 kB 00:01 2024-03-25T18:31:09.519 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:09.883 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:09.901 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:09.954 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:10.021 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:10.151 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:10.211 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:10.231 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:10.266 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:10.408 INFO:teuthology.orchestra.run.smithi193.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 177 kB/s | 221 kB 00:01 2024-03-25T18:31:10.440 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:10.450 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 281 kB/s | 298 kB 00:01 2024-03-25T18:31:10.475 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:10.555 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:10.637 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:10.654 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:10.705 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:10.738 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:10.750 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 331 kB/s | 99 kB 00:00 2024-03-25T18:31:10.773 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:10.827 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:10.844 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:10.861 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:10.875 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 850 kB/s | 106 kB 00:00 2024-03-25T18:31:10.900 INFO:teuthology.orchestra.run.smithi193.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 73 kB/s | 36 kB 00:00 2024-03-25T18:31:10.926 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:31:10.932 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:10.968 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T18:31:10.969 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:11.043 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T18:31:11.048 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:11.130 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:11.146 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.7 MB/s | 282 kB 00:00 2024-03-25T18:31:11.150 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:11.205 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:11.238 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:11.272 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:11.324 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:11.341 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:11.358 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:11.400 INFO:teuthology.orchestra.run.smithi193.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 95 kB/s | 47 kB 00:00 2024-03-25T18:31:11.437 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 141 kB/s | 41 kB 00:00 2024-03-25T18:31:11.455 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:11.510 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:11.546 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:11.624 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:11.755 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:11.800 INFO:teuthology.orchestra.run.smithi193.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 772 kB/s | 309 kB 00:00 2024-03-25T18:31:11.842 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:11.991 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:12.142 INFO:teuthology.orchestra.run.smithi193.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-25T18:31:12.156 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:12.234 INFO:teuthology.orchestra.run.smithi193.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-25T18:31:12.309 INFO:teuthology.orchestra.run.smithi193.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 649 kB/s | 49 kB 00:00 2024-03-25T18:31:12.376 INFO:teuthology.orchestra.run.smithi193.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T18:31:12.468 INFO:teuthology.orchestra.run.smithi193.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-25T18:31:12.511 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:12.568 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:12.577 INFO:teuthology.orchestra.run.smithi193.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-25T18:31:12.593 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:12.739 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:12.761 INFO:teuthology.orchestra.run.smithi193.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-25T18:31:12.827 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:12.952 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 146 kB/s | 221 kB 00:01 2024-03-25T18:31:12.998 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.064 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.081 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:13.113 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:13.145 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:13.178 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:13.210 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:13.290 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:13.370 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:13.429 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.446 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.606 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:13.695 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:13.727 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 46 kB/s | 36 kB 00:00 2024-03-25T18:31:13.745 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.800 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.817 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:13.851 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:13.884 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:13.916 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:13.948 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:14.028 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:14.108 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:14.162 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.179 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.319 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:14.407 INFO:teuthology.orchestra.run.smithi071.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:14.436 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 67 kB/s | 47 kB 00:00 2024-03-25T18:31:14.544 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-03-25T18:31:14.636 INFO:teuthology.orchestra.run.smithi193.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 424 kB/s | 5.1 MB 00:12 2024-03-25T18:31:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-25T18:31:14.803 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 781 kB/s | 26 kB 00:00 2024-03-25T18:31:14.854 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-03-25T18:31:14.887 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-25T18:31:14.979 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-25T18:31:15.021 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-03-25T18:31:15.138 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-25T18:31:16.630 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 622 kB/s | 20 MB 00:32 2024-03-25T18:31:18.096 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 323 kB/s | 5.1 MB 00:16 2024-03-25T18:31:20.936 INFO:teuthology.orchestra.run.smithi193.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 526 kB/s | 20 MB 00:38 2024-03-25T18:31:22.635 INFO:teuthology.orchestra.run.smithi193.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 401 kB/s | 3.1 MB 00:07 2024-03-25T18:31:22.720 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 526 kB/s | 3.1 MB 00:06 2024-03-25T18:31:23.702 INFO:teuthology.orchestra.run.smithi193.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 316 kB/s | 3.4 MB 00:10 2024-03-25T18:31:23.709 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:23.709 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.4 MB/s | 58 MB 00:41 2024-03-25T18:31:23.745 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:31:23.783 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:31:23.783 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:31:24.267 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:31:24.267 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:31:25.761 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:31:25.840 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:31:25.882 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:31:25.907 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:31:25.930 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:25.948 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:26.139 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:26.194 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:26.320 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 309 kB/s | 3.4 MB 00:11 2024-03-25T18:31:26.327 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:26.327 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.4 MB/s | 58 MB 00:42 2024-03-25T18:31:26.359 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:31:26.398 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:31:26.398 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:31:26.510 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:31:26.567 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:26.745 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:31:26.768 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:26.823 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:26.851 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:31:26.881 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:31:26.881 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:31:26.895 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:31:26.963 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:26.988 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:27.035 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:27.069 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:31:27.175 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:31:27.220 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:31:27.261 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:31:27.302 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:31:27.330 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:31:27.372 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:31:27.679 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:31:27.710 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:27.791 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:27.823 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:31:27.865 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:31:28.321 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:31:28.341 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:31:28.346 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:28.401 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:28.417 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:31:28.457 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:31:28.477 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:31:28.483 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:31:28.506 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:31:28.514 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:28.530 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:28.739 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:28.794 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:29.114 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:31:29.177 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:29.268 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:29.355 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:31:29.378 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:29.440 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:29.469 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:31:29.513 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:31:29.580 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:29.605 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:29.652 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:29.687 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:31:29.800 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:31:29.838 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:31:29.879 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:31:29.916 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:31:29.940 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:31:29.973 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:31:30.304 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:31:30.336 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:30.426 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:31:30.502 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:31:30.708 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:30.753 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:30.973 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:31:30.997 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:31.068 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:31.135 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:31:31.166 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:31:31.310 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:31.348 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:31.439 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:31.439 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:31:31.440 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:31:31.440 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:31:31.474 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:33.948 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:34.027 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:34.027 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:31:34.027 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:31:34.027 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:31:34.058 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:40.661 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:40.661 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-03-25T18:31:40.661 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-03-25T18:31:40.661 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-03-25T18:31:40.662 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-03-25T18:31:40.662 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-03-25T18:31:40.662 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-03-25T18:31:40.662 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-03-25T18:31:40.662 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:31:41.582 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:41.607 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:41.607 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:31:41.607 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:31:41.607 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:41.607 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:41.607 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:31:41.609 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:42.909 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:42.909 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:31:42.909 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:31:42.909 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:31:42.909 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:31:42.910 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:31:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:31:42.912 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:31:42.912 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:31:42.912 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:31:42.912 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:42.912 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:31:42.912 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:31:43.399 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:43.399 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-03-25T18:31:43.399 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-03-25T18:31:43.399 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-03-25T18:31:43.399 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-03-25T18:31:43.399 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-03-25T18:31:43.399 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-03-25T18:31:43.400 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-03-25T18:31:43.400 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:31:43.883 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:43.883 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.886 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:43.886 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:31:43.886 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:31:43.886 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:31:43.886 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:31:43.886 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:31:43.886 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:31:43.886 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:31:44.240 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-test 2024-03-25T18:31:44.295 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:44.318 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:44.319 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:31:44.319 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:31:44.319 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:44.319 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:44.319 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:31:44.321 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:44.414 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:44.414 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:44.794 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:05 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:31:44.945 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:31:44.946 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:31:44.946 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:31:44.946 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-03-25T18:31:44.947 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:31:44.948 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 48 M 2024-03-25T18:31:44.948 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 204 M 2024-03-25T18:31:44.948 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:31:45.221 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T18:31:45.313 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.3 MB/s | 300 kB 00:00 2024-03-25T18:31:45.355 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T18:31:45.421 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-03-25T18:31:45.504 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:45.505 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:31:45.506 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:31:45.507 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:31:45.507 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:31:45.507 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:45.507 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:31:45.507 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:31:46.197 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 146 kB/s | 163 kB 00:01 2024-03-25T18:31:46.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:46.645 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:31:46.645 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-03-25T18:31:46.645 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.645 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.645 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.646 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:31:46.647 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:31:46.648 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:31:46.648 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:31:46.648 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:31:46.982 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2024-03-25T18:31:47.533 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:06 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:31:47.681 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:31:47.682 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:31:47.683 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:31:47.683 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:31:47.683 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:31:47.683 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-03-25T18:31:47.683 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:31:47.683 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 48 M 2024-03-25T18:31:47.683 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 204 M 2024-03-25T18:31:47.683 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:31:47.941 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T18:31:48.033 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.2 MB/s | 300 kB 00:00 2024-03-25T18:31:48.075 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T18:31:48.358 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 662 kB/s | 187 kB 00:00 2024-03-25T18:31:48.467 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 243 kB/s | 163 kB 00:00 2024-03-25T18:33:42.781 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 411 kB/s | 47 MB 01:57 2024-03-25T18:33:42.782 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:42.783 INFO:teuthology.orchestra.run.smithi193.stdout:Total 419 kB/s | 48 MB 01:57 2024-03-25T18:33:42.794 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:33:42.813 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:33:42.813 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:33:43.193 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:33:43.193 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:33:43.788 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:33:43.846 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:33:43.885 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:33:43.925 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:43.971 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:33:43.999 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:48.462 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:48.551 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:33:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:33:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:33:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:33:51.236 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:33:51.236 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:33:51.236 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:33:51.237 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:51.237 INFO:teuthology.orchestra.run.smithi193.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:33:51.237 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:51.237 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:33:51.237 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:33:51.237 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:33:51.237 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:33:51.237 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:33:51.352 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 392 kB/s | 47 MB 02:03 2024-03-25T18:33:51.354 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:51.354 INFO:teuthology.orchestra.run.smithi071.stdout:Total 399 kB/s | 48 MB 02:03 2024-03-25T18:33:51.365 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:33:51.384 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:33:51.384 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:33:51.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph 2024-03-25T18:33:51.765 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:33:51.765 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:33:52.112 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:13 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:33:52.264 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:33:52.266 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-25T18:33:52.266 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:33:52.266 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:33:52.267 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:33:52.268 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:33:52.269 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-25T18:33:52.270 INFO:teuthology.orchestra.run.smithi193.stdout:Install 36 Packages 2024-03-25T18:33:52.270 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:33:52.270 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 30 M 2024-03-25T18:33:52.270 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 107 M 2024-03-25T18:33:52.270 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:33:52.353 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:33:52.427 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:33:52.470 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:33:52.510 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:52.561 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:33:52.588 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:53.027 INFO:teuthology.orchestra.run.smithi193.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:33:56.519 INFO:teuthology.orchestra.run.smithi193.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 387 kB/s | 1.5 MB 00:03 2024-03-25T18:33:57.155 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:57.229 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:58.277 INFO:teuthology.orchestra.run.smithi193.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 382 kB/s | 2.1 MB 00:05 2024-03-25T18:33:59.144 INFO:teuthology.orchestra.run.smithi193.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 285 kB/s | 247 kB 00:00 2024-03-25T18:33:59.261 INFO:teuthology.orchestra.run.smithi193.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:33:59.303 INFO:teuthology.orchestra.run.smithi193.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:33:59.328 INFO:teuthology.orchestra.run.smithi193.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:33:59.354 INFO:teuthology.orchestra.run.smithi193.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-25T18:33:59.379 INFO:teuthology.orchestra.run.smithi193.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:33:59.454 INFO:teuthology.orchestra.run.smithi193.stdout:(10/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 746 kB/s | 4.7 MB 00:06 2024-03-25T18:33:59.479 INFO:teuthology.orchestra.run.smithi193.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2024-03-25T18:33:59.527 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:59.527 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:33:59.527 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:33:59.527 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:59.528 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:33:59.528 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:33:59.530 INFO:teuthology.orchestra.run.smithi193.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:33:59.555 INFO:teuthology.orchestra.run.smithi193.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-25T18:33:59.580 INFO:teuthology.orchestra.run.smithi193.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T18:33:59.605 INFO:teuthology.orchestra.run.smithi193.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T18:33:59.722 INFO:teuthology.orchestra.run.smithi193.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-25T18:33:59.790 INFO:teuthology.orchestra.run.smithi193.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 598 kB/s | 125 kB 00:00 2024-03-25T18:33:59.857 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:33:59.858 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:34:00.166 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2024-03-25T18:34:00.223 INFO:teuthology.orchestra.run.smithi193.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 100 kB/s | 43 kB 00:00 2024-03-25T18:34:00.307 INFO:teuthology.orchestra.run.smithi193.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 259 kB/s | 151 kB 00:00 2024-03-25T18:34:00.532 INFO:teuthology.orchestra.run.smithi193.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 557 kB/s | 172 kB 00:00 2024-03-25T18:34:00.557 INFO:teuthology.orchestra.run.smithi193.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-25T18:34:00.641 INFO:teuthology.orchestra.run.smithi193.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-03-25T18:34:00.666 INFO:teuthology.orchestra.run.smithi193.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-03-25T18:34:00.728 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:03:19 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:34:00.750 INFO:teuthology.orchestra.run.smithi193.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-25T18:34:00.775 INFO:teuthology.orchestra.run.smithi193.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-25T18:34:00.859 INFO:teuthology.orchestra.run.smithi193.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 180 kB/s | 20 kB 00:00 2024-03-25T18:34:00.887 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:34:00.889 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-25T18:34:00.889 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:34:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:34:00.891 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout:Install 36 Packages 2024-03-25T18:34:00.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:00.893 INFO:teuthology.orchestra.run.smithi193.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 394 kB/s | 46 kB 00:00 2024-03-25T18:34:00.893 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 M 2024-03-25T18:34:00.893 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 107 M 2024-03-25T18:34:00.893 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:34:00.993 INFO:teuthology.orchestra.run.smithi193.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 590 kB/s | 79 kB 00:00 2024-03-25T18:34:01.052 INFO:teuthology.orchestra.run.smithi193.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-25T18:34:01.127 INFO:teuthology.orchestra.run.smithi193.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2024-03-25T18:34:01.185 INFO:teuthology.orchestra.run.smithi193.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 673 kB/s | 90 kB 00:00 2024-03-25T18:34:01.244 INFO:teuthology.orchestra.run.smithi193.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 306 kB/s | 36 kB 00:00 2024-03-25T18:34:01.336 INFO:teuthology.orchestra.run.smithi193.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-03-25T18:34:01.454 INFO:teuthology.orchestra.run.smithi193.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-03-25T18:34:01.472 INFO:teuthology.orchestra.run.smithi071.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:34:01.495 INFO:teuthology.orchestra.run.smithi193.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-03-25T18:34:05.963 INFO:teuthology.orchestra.run.smithi071.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 309 kB/s | 1.5 MB 00:04 2024-03-25T18:34:11.346 INFO:teuthology.orchestra.run.smithi071.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 211 kB/s | 2.1 MB 00:10 2024-03-25T18:34:12.120 INFO:teuthology.orchestra.run.smithi071.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 319 kB/s | 247 kB 00:00 2024-03-25T18:34:12.237 INFO:teuthology.orchestra.run.smithi071.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:34:12.296 INFO:teuthology.orchestra.run.smithi071.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T18:34:12.329 INFO:teuthology.orchestra.run.smithi071.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-25T18:34:12.388 INFO:teuthology.orchestra.run.smithi071.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 812 kB/s | 47 kB 00:00 2024-03-25T18:34:12.430 INFO:teuthology.orchestra.run.smithi071.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 924 kB/s | 39 kB 00:00 2024-03-25T18:34:12.472 INFO:teuthology.orchestra.run.smithi071.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:34:12.548 INFO:teuthology.orchestra.run.smithi071.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:34:12.581 INFO:teuthology.orchestra.run.smithi071.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:34:12.623 INFO:teuthology.orchestra.run.smithi071.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:34:12.656 INFO:teuthology.orchestra.run.smithi071.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T18:34:12.849 INFO:teuthology.orchestra.run.smithi071.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 651 kB/s | 125 kB 00:00 2024-03-25T18:34:13.182 INFO:teuthology.orchestra.run.smithi071.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 67 kB/s | 22 kB 00:00 2024-03-25T18:34:13.291 INFO:teuthology.orchestra.run.smithi071.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-03-25T18:34:13.324 INFO:teuthology.orchestra.run.smithi071.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-25T18:34:13.375 INFO:teuthology.orchestra.run.smithi071.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-03-25T18:34:13.425 INFO:teuthology.orchestra.run.smithi071.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 7.0 MB/s | 358 kB 00:00 2024-03-25T18:34:13.458 INFO:teuthology.orchestra.run.smithi071.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 323 kB/s | 11 kB 00:00 2024-03-25T18:34:13.492 INFO:teuthology.orchestra.run.smithi071.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 531 kB/s | 18 kB 00:00 2024-03-25T18:34:13.526 INFO:teuthology.orchestra.run.smithi071.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 695 kB/s | 23 kB 00:00 2024-03-25T18:34:13.559 INFO:teuthology.orchestra.run.smithi071.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 585 kB/s | 19 kB 00:00 2024-03-25T18:34:13.593 INFO:teuthology.orchestra.run.smithi071.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 591 kB/s | 20 kB 00:00 2024-03-25T18:34:13.627 INFO:teuthology.orchestra.run.smithi071.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-03-25T18:34:13.652 INFO:teuthology.orchestra.run.smithi071.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 3.1 MB/s | 79 kB 00:00 2024-03-25T18:34:13.685 INFO:teuthology.orchestra.run.smithi071.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-03-25T18:34:13.719 INFO:teuthology.orchestra.run.smithi071.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 495 kB/s | 16 kB 00:00 2024-03-25T18:34:13.753 INFO:teuthology.orchestra.run.smithi071.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-03-25T18:34:13.786 INFO:teuthology.orchestra.run.smithi071.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 1.1 MB/s | 36 kB 00:00 2024-03-25T18:34:13.836 INFO:teuthology.orchestra.run.smithi071.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T18:34:13.895 INFO:teuthology.orchestra.run.smithi071.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-25T18:34:13.928 INFO:teuthology.orchestra.run.smithi071.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 605 kB/s | 20 kB 00:00 2024-03-25T18:34:16.428 INFO:teuthology.orchestra.run.smithi071.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 321 kB/s | 4.7 MB 00:14 2024-03-25T18:34:34.001 INFO:teuthology.orchestra.run.smithi071.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 613 kB/s | 17 MB 00:28 2024-03-25T18:34:34.004 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:34.005 INFO:teuthology.orchestra.run.smithi071.stdout:Total 917 kB/s | 30 MB 00:33 2024-03-25T18:34:34.027 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:34:34.069 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:34:34.069 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:34:34.476 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:34:34.476 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:34:35.805 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:34:35.861 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:34:35.913 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:34:35.947 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:34:36.018 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:34:36.062 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:34:36.095 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:34:36.187 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:34:36.281 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:34:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:34:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:34:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:34:36.495 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:34:36.561 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:34:36.615 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:34:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:34:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:34:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:34:36.933 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:34:36.977 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:34:37.052 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:34:37.087 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:37.129 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:34:37.169 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:34:37.204 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:34:37.239 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:34:37.329 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:34:37.369 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:34:37.443 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:34:37.577 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:34:37.702 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:37.873 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:34:37.899 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:37.924 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:37.924 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:37.924 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:34:37.924 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:37.924 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:37.924 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:39.383 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:39.409 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:39.409 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:39.409 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:34:39.409 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:39.409 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:39.409 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:39.859 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:39.882 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:39.882 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:39.882 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:34:39.882 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:39.882 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:39.883 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:40.093 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:40.116 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:40.116 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:40.116 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:34:40.116 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:40.116 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:40.116 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:40.182 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:40.681 INFO:teuthology.orchestra.run.smithi193.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 389 kB/s | 17 MB 00:44 2024-03-25T18:34:40.687 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:40.687 INFO:teuthology.orchestra.run.smithi193.stdout:Total 627 kB/s | 30 MB 00:48 2024-03-25T18:34:40.711 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:34:40.754 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:34:40.755 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:34:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:34:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:34:41.873 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:41.873 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:34:41.873 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:34:41.874 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:34:41.875 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:34:41.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:34:41.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:34:41.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:41.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:34:41.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:34:41.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:34:41.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:34:41.876 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:34:42.502 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:34:42.550 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:34:42.589 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:34:42.619 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:34:42.688 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:34:42.726 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:34:42.777 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:42.806 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:34:42.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:34:42.808 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:34:42.809 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:34:42.809 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:34:42.809 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:42.809 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:34:42.876 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:34:42.954 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:34:42.996 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:34:43.066 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:34:43.121 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:34:43.160 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:34:43.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2024-03-25T18:34:43.208 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:34:43.263 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:34:43.446 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:34:43.522 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:34:43.553 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:34:43.587 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:34:43.617 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:34:43.693 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:34:43.728 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:43.732 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:02 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:34:43.761 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:34:43.801 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:34:43.828 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:43.836 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:34:43.871 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:34:43.880 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:34:43.881 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:34:43.881 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:34:43.947 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2024-03-25T18:34:43.960 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:34:44.002 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:34:44.073 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:34:44.210 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:34:44.343 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:44.502 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:34:44.507 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:03 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:34:44.532 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:44.558 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:44.559 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:44.559 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:34:44.559 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:44.559 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:44.559 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:44.655 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:44.657 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 747 k 2024-03-25T18:34:44.657 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 754 k 2024-03-25T18:34:44.657 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:34:46.017 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:46.044 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:46.044 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:46.044 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:34:46.044 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:46.044 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:46.044 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:46.510 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:46.534 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:46.534 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:46.534 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:34:46.535 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:46.535 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:46.535 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:46.744 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:46.768 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:46.768 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:46.768 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:34:46.768 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:46.768 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:46.768 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:46.835 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:47.105 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 305 kB/s | 747 kB 00:02 2024-03-25T18:34:47.106 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:47.106 INFO:teuthology.orchestra.run.smithi071.stdout:Total 305 kB/s | 747 kB 00:02 2024-03-25T18:34:47.106 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:34:47.112 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:34:47.113 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:34:47.119 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:34:47.119 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:34:47.195 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:34:47.575 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:47.677 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:48.075 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:48.378 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:48.378 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:48.379 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:34:48.379 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:48.379 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:48.379 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:34:48.541 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:34:48.542 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:34:48.543 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:34:48.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:34:49.164 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:08 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:34:49.317 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:34:49.318 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:34:49.319 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:49.319 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 145 k 2024-03-25T18:34:49.319 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 436 k 2024-03-25T18:34:49.319 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:49.643 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:34:49.644 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:34:49.645 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:34:49.646 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:34:49.646 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:34:49.646 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:34:49.646 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:49.646 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:34:49.952 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 230 kB/s | 145 kB 00:00 2024-03-25T18:34:49.953 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:49.953 INFO:teuthology.orchestra.run.smithi071.stdout:Total 229 kB/s | 145 kB 00:00 2024-03-25T18:34:49.953 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:34:49.961 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:34:49.961 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:34:49.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base 2024-03-25T18:34:50.013 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:34:50.013 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:34:50.147 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:34:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:50.244 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:50.244 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:34:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:50.541 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:34:50.640 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:50.691 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:34:50.692 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:34:50.693 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:34:50.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install cephadm 2024-03-25T18:34:51.246 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:51.247 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:51.247 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:34:51.247 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:51.247 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:51.247 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:34:51.310 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:12 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:34:51.458 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:34:51.459 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:34:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:34:51.459 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:34:51.459 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:34:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:34:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:51.460 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:34:51.460 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:34:51.460 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:34:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:51.460 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 747 k 2024-03-25T18:34:51.460 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 754 k 2024-03-25T18:34:51.460 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:34:51.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2024-03-25T18:34:52.048 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:34:52.144 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:52.196 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:34:52.198 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:34:52.198 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:34:52.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:34:52.822 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:34:52.970 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:34:52.971 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-25T18:34:52.971 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:34:52.971 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-25T18:34:52.971 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:34:52.971 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:34:52.971 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:34:52.971 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-25T18:34:52.972 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-03-25T18:34:52.973 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:34:52.973 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.9 M 2024-03-25T18:34:52.973 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-03-25T18:34:52.973 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:34:53.630 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T18:34:53.731 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 55 kB/s | 31 kB 00:00 2024-03-25T18:34:53.789 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-25T18:34:53.814 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T18:34:53.998 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-03-25T18:34:54.223 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-03-25T18:34:54.248 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T18:34:54.324 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.9 MB/s | 144 kB 00:00 2024-03-25T18:34:54.391 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 462 kB/s | 31 kB 00:00 2024-03-25T18:34:54.466 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-25T18:34:54.591 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.5 MB/s | 2.0 MB 00:00 2024-03-25T18:34:54.715 INFO:teuthology.orchestra.run.smithi193.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 230 kB/s | 747 kB 00:03 2024-03-25T18:34:54.716 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:54.716 INFO:teuthology.orchestra.run.smithi193.stdout:Total 229 kB/s | 747 kB 00:03 2024-03-25T18:34:54.716 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:34:54.722 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:34:54.722 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:34:54.728 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:34:54.729 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:34:54.809 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:34:55.294 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:55.380 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:55.732 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:56.008 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:56.008 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:56.009 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:34:56.009 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:56.009 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:56.009 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:34:56.232 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:34:56.807 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:34:56.961 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:34:56.961 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:34:56.961 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:34:56.961 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:34:56.962 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:34:56.962 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:34:56.962 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:56.962 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:34:56.962 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:34:56.962 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:34:56.962 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:56.963 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 145 k 2024-03-25T18:34:56.963 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 436 k 2024-03-25T18:34:56.963 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:34:57.798 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 174 kB/s | 145 kB 00:00 2024-03-25T18:34:57.798 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:57.799 INFO:teuthology.orchestra.run.smithi193.stdout:Total 174 kB/s | 145 kB 00:00 2024-03-25T18:34:57.799 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:34:57.807 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:34:57.807 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:34:57.861 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:34:57.861 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:34:58.004 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:34:58.074 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:58.098 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:58.098 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:58.099 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:34:58.099 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:59.084 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:59.084 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:59.084 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:34:59.084 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:59.084 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:34:59.084 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:34:59.354 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr 2024-03-25T18:34:59.925 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:00.027 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:00.078 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:00.079 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:35:00.079 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:00.149 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:35:00.582 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 246 kB/s | 1.8 MB 00:07 2024-03-25T18:35:00.588 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:00.588 INFO:teuthology.orchestra.run.smithi071.stdout:Total 922 kB/s | 6.9 MB 00:07 2024-03-25T18:35:00.607 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:00.626 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:00.626 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:00.708 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:21 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:00.801 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:00.802 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:00.857 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:00.858 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-25T18:35:00.858 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:35:00.859 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:00.860 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:00.860 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-25T18:35:00.860 INFO:teuthology.orchestra.run.smithi193.stdout:Install 12 Packages 2024-03-25T18:35:00.860 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:00.860 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 6.9 M 2024-03-25T18:35:00.860 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 26 M 2024-03-25T18:35:00.860 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:35:01.493 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:35:01.510 INFO:teuthology.orchestra.run.smithi193.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 30 kB/s | 15 kB 00:00 2024-03-25T18:35:01.539 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:35:01.570 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:35:01.661 INFO:teuthology.orchestra.run.smithi193.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T18:35:01.680 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:35:01.686 INFO:teuthology.orchestra.run.smithi193.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 45 kB/s | 31 kB 00:00 2024-03-25T18:35:01.720 INFO:teuthology.orchestra.run.smithi193.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.0 MB/s | 238 kB 00:00 2024-03-25T18:35:01.804 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:35:01.853 INFO:teuthology.orchestra.run.smithi193.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.1 MB/s | 863 kB 00:00 2024-03-25T18:35:01.904 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:35:01.948 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:35:01.986 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:02.018 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:35:02.037 INFO:teuthology.orchestra.run.smithi193.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-03-25T18:35:02.261 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:35:02.292 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:02.396 INFO:teuthology.orchestra.run.smithi193.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 811 kB/s | 548 kB 00:00 2024-03-25T18:35:02.463 INFO:teuthology.orchestra.run.smithi193.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 4.8 MB/s | 2.0 MB 00:00 2024-03-25T18:35:02.555 INFO:teuthology.orchestra.run.smithi193.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 910 kB/s | 144 kB 00:00 2024-03-25T18:35:02.572 INFO:teuthology.orchestra.run.smithi193.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-25T18:35:02.730 INFO:teuthology.orchestra.run.smithi193.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-03-25T18:35:03.088 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:03.089 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:35:03.090 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:35:03.436 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:35:03.436 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:03.436 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:03.436 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:35:03.437 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:35:03.438 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:03.438 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:03.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:35:04.233 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:04.384 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:04.385 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-25T18:35:04.385 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:35:04.385 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-25T18:35:04.385 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:04.385 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:35:04.385 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:35:04.385 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:35:04.385 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-25T18:35:04.386 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-03-25T18:35:04.387 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:04.387 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 37 M 2024-03-25T18:35:04.387 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 209 M 2024-03-25T18:35:04.387 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:35:04.422 INFO:teuthology.orchestra.run.smithi193.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 532 kB/s | 1.8 MB 00:03 2024-03-25T18:35:04.426 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:04.426 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.9 MB/s | 6.9 MB 00:03 2024-03-25T18:35:04.446 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:04.465 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:04.465 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:04.637 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:04.638 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:04.746 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 283 kB/s | 32 kB 00:00 2024-03-25T18:35:04.796 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 651 kB/s | 17 kB 00:00 2024-03-25T18:35:04.830 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:35:05.022 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T18:35:05.163 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T18:35:05.171 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:05.197 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 6.5 MB/s | 217 kB 00:00 2024-03-25T18:35:05.255 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:35:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:35:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:35:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:35:05.422 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2024-03-25T18:35:05.488 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:35:05.619 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:35:05.623 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-25T18:35:05.714 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:35:05.748 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-25T18:35:05.765 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:35:05.773 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-25T18:35:05.819 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:05.850 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:35:05.990 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-25T18:35:06.093 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:35:06.125 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:35:06.949 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:35:06.950 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:06.950 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:35:06.950 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:35:07.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:35:07.300 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:07.300 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:07.300 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:35:07.300 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:07.300 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:07.300 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:07.301 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:07.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:35:08.110 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:29 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:08.264 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:08.265 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-25T18:35:08.265 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:35:08.265 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-25T18:35:08.265 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:08.265 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:35:08.266 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:08.267 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:08.267 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-25T18:35:08.267 INFO:teuthology.orchestra.run.smithi193.stdout:Install 12 Packages 2024-03-25T18:35:08.267 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:08.267 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 37 M 2024-03-25T18:35:08.267 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 209 M 2024-03-25T18:35:08.267 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:08.555 INFO:teuthology.orchestra.run.smithi193.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 254 kB/s | 32 kB 00:00 2024-03-25T18:35:08.613 INFO:teuthology.orchestra.run.smithi193.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 481 kB/s | 17 kB 00:00 2024-03-25T18:35:08.663 INFO:teuthology.orchestra.run.smithi193.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2024-03-25T18:35:08.822 INFO:teuthology.orchestra.run.smithi193.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T18:35:08.956 INFO:teuthology.orchestra.run.smithi193.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:35:09.031 INFO:teuthology.orchestra.run.smithi193.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-03-25T18:35:09.056 INFO:teuthology.orchestra.run.smithi193.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-25T18:35:09.365 INFO:teuthology.orchestra.run.smithi193.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-25T18:35:09.874 INFO:teuthology.orchestra.run.smithi193.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-03-25T18:35:09.907 INFO:teuthology.orchestra.run.smithi193.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 946 kB/s | 803 kB 00:00 2024-03-25T18:35:10.066 INFO:teuthology.orchestra.run.smithi193.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-25T18:35:17.923 INFO:teuthology.orchestra.run.smithi193.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 797 kB/s | 7.4 MB 00:09 2024-03-25T18:35:17.926 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:17.926 INFO:teuthology.orchestra.run.smithi193.stdout:Total 3.8 MB/s | 37 MB 00:09 2024-03-25T18:35:17.970 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:17.996 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:17.996 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:18.354 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:18.355 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:19.067 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:19.113 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 523 kB/s | 7.4 MB 00:14 2024-03-25T18:35:19.115 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:19.116 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.5 MB/s | 37 MB 00:14 2024-03-25T18:35:19.129 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:35:19.160 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:19.160 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:35:19.186 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:19.186 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:19.244 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:19.297 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:35:19.466 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:19.505 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:19.537 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:19.538 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:19.584 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:35:20.252 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:20.259 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:35:20.327 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:35:20.367 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:35:20.367 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:35:20.468 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:20.504 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:35:20.682 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:20.710 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:20.795 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:35:21.435 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:35:21.549 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:35:21.602 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:35:22.633 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:35:22.664 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:22.781 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:35:24.021 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:35:24.059 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:25.299 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:25.299 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:35:25.300 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:35:25.740 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:35:25.741 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:35:25.741 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:35:25.741 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:35:25.741 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:25.741 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:25.741 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:35:25.741 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:25.741 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:26.119 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:35:26.645 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:35:26.646 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:35:26.684 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:26.834 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:26.835 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:35:26.835 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:35:26.835 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:35:26.835 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:26.835 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:35:26.836 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:35:26.837 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:35:26.837 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:35:26.837 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:26.837 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:26.837 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:35:26.837 INFO:teuthology.orchestra.run.smithi193.stdout:Install 13 Packages 2024-03-25T18:35:26.837 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:26.837 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 2.0 M 2024-03-25T18:35:26.838 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 26 M 2024-03-25T18:35:26.838 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:27.074 INFO:teuthology.orchestra.run.smithi193.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 263 kB/s | 24 kB 00:00 2024-03-25T18:35:27.125 INFO:teuthology.orchestra.run.smithi193.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-25T18:35:27.133 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:35:27.133 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:27.133 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:27.133 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:27.133 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:27.134 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:27.166 INFO:teuthology.orchestra.run.smithi193.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T18:35:27.191 INFO:teuthology.orchestra.run.smithi193.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:35:27.216 INFO:teuthology.orchestra.run.smithi193.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:35:27.242 INFO:teuthology.orchestra.run.smithi193.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2024-03-25T18:35:27.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:35:27.484 INFO:teuthology.orchestra.run.smithi193.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-25T18:35:27.509 INFO:teuthology.orchestra.run.smithi193.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-25T18:35:27.535 INFO:teuthology.orchestra.run.smithi193.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 94 kB/s | 52 kB 00:00 2024-03-25T18:35:27.660 INFO:teuthology.orchestra.run.smithi193.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-03-25T18:35:27.693 INFO:teuthology.orchestra.run.smithi193.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2024-03-25T18:35:27.786 INFO:teuthology.orchestra.run.smithi193.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-25T18:35:27.811 INFO:teuthology.orchestra.run.smithi193.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-03-25T18:35:27.814 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:27.814 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-25T18:35:27.814 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:27.826 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:27.827 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:27.917 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:27.918 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:28.043 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:28.193 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:35:28.195 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2024-03-25T18:35:28.196 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:28.197 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.0 M 2024-03-25T18:35:28.197 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-03-25T18:35:28.197 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:35:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:28.467 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 156 kB/s | 18 kB 00:00 2024-03-25T18:35:28.493 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 24 kB 00:00 2024-03-25T18:35:28.534 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T18:35:28.559 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:35:28.585 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:35:28.610 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-25T18:35:28.614 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:35:28.635 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 184 kB/s | 52 kB 00:00 2024-03-25T18:35:28.670 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:35:28.726 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:35:28.763 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:35:28.804 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:35:28.931 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:35:28.969 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 42 kB/s | 15 kB 00:00 2024-03-25T18:35:28.994 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 79 kB/s | 32 kB 00:00 2024-03-25T18:35:29.041 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:35:29.089 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:35:29.211 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 272 kB/s | 59 kB 00:00 2024-03-25T18:35:29.237 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 337 kB/s | 203 kB 00:00 2024-03-25T18:35:29.379 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2024-03-25T18:35:29.462 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-03-25T18:35:29.466 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:29.466 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-25T18:35:29.467 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:29.480 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:29.480 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:29.570 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:29.572 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:29.811 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:35:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:35:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:35:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:35:29.965 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:30.077 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:30.241 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:35:30.296 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:35:30.351 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:35:30.388 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:35:30.430 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:35:30.532 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:35:30.641 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:35:30.698 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:35:30.728 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:30.728 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:35:30.728 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:35:30.728 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:35:30.728 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:35:30.729 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:35:30.729 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:35:30.729 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:35:30.729 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:35:30.729 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:35:30.729 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:35:30.729 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:35:30.729 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:35:31.097 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:35:31.098 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:35:31.098 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:35:31.098 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:35:31.098 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:35:31.098 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:35:31.098 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:35:31.098 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:31.098 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:31.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:35:31.414 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:35:31.470 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:35:31.503 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:35:31.552 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:35:31.574 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:31.904 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:32.055 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:32.055 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-03-25T18:35:32.056 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:32.057 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 6.8 M 2024-03-25T18:35:32.057 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 33 M 2024-03-25T18:35:32.057 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:32.373 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:35:32.374 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:35:32.513 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.1 MB/s | 248 kB 00:00 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:35:32.732 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:35:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:35:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:35:32.733 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:32.733 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:32.764 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-25T18:35:32.789 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 352 kB/s | 138 kB 00:00 2024-03-25T18:35:32.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:35:33.089 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 177 kB/s | 58 kB 00:00 2024-03-25T18:35:33.114 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 157 kB/s | 51 kB 00:00 2024-03-25T18:35:33.140 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 910 kB/s | 548 kB 00:00 2024-03-25T18:35:33.141 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:33.141 INFO:teuthology.orchestra.run.smithi193.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2024-03-25T18:35:33.154 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:33.165 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:33.165 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:33.255 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:33.256 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:33.552 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:33.559 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:33.710 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:33.710 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:35:33.711 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-03-25T18:35:33.712 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:33.712 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.8 M 2024-03-25T18:35:33.712 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 33 M 2024-03-25T18:35:33.712 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:35:33.713 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:35:33.758 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:35:33.979 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.2 MB/s | 248 kB 00:00 2024-03-25T18:35:34.173 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-25T18:35:34.288 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:35:34.379 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:35:34.404 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 248 kB/s | 58 kB 00:00 2024-03-25T18:35:34.440 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:35:34.471 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:34.488 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T18:35:34.514 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 470 kB/s | 51 kB 00:00 2024-03-25T18:35:34.539 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 206 kB/s | 138 kB 00:00 2024-03-25T18:35:34.541 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:34.541 INFO:teuthology.orchestra.run.smithi071.stdout:Total 8.2 MB/s | 6.8 MB 00:00 2024-03-25T18:35:34.554 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:34.564 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:34.564 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:34.653 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:34.653 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:34.955 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:35:35.165 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:35:35.315 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:35.315 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:35:35.315 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:35:35.315 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:35:35.315 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:35:35.316 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:35:35.618 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:35:35.618 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:35.618 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:35.618 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:35.618 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:35:35.619 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:35:35.619 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:35:35.619 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:35:35.619 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:35:35.619 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:35.619 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:35.687 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:35:35.768 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:35:35.839 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:35:35.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-fuse 2024-03-25T18:35:35.869 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:36.405 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:04:57 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:36.557 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:36.557 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:36.557 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:35:36.557 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:36.557 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:36.558 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:35:36.558 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:35:36.558 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:35:36.558 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:36.558 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:36.558 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:36.558 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-03-25T18:35:36.558 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:36.559 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 964 k 2024-03-25T18:35:36.559 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.8 M 2024-03-25T18:35:36.559 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:36.674 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:36.674 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:35:36.674 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:35:36.674 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:35:36.674 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:35:36.674 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:35:36.677 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 663 kB/s | 79 kB 00:00 2024-03-25T18:35:36.970 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:35:36.970 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:36.970 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:36.970 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:36.970 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:35:36.970 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:35:36.970 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:35:36.970 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:35:36.971 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:35:36.971 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:36.971 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:37.189 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2024-03-25T18:35:37.749 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:56 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:37.899 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:37.899 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:37.899 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-25T18:35:37.900 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:37.901 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 964 k 2024-03-25T18:35:37.901 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.8 M 2024-03-25T18:35:37.901 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:35:38.006 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 748 kB/s | 79 kB 00:00 2024-03-25T18:35:38.402 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 480 kB/s | 885 kB 00:01 2024-03-25T18:35:38.404 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:38.405 INFO:teuthology.orchestra.run.smithi193.stdout:Total 522 kB/s | 964 kB 00:01 2024-03-25T18:35:38.405 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:38.428 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:38.428 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:38.505 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:38.505 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:38.683 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:38.812 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:35:38.885 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:39.604 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:39.604 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:35:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:35:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:39.928 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:35:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:39.928 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:40.014 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 419 kB/s | 885 kB 00:02 2024-03-25T18:35:40.016 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:40.017 INFO:teuthology.orchestra.run.smithi071.stdout:Total 455 kB/s | 964 kB 00:02 2024-03-25T18:35:40.017 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:40.040 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:40.040 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:40.116 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:40.116 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:40.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-volume 2024-03-25T18:35:40.296 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:40.433 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:35:40.506 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:40.827 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:40.977 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:40.978 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:40.979 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-03-25T18:35:40.979 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:40.979 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 332 k 2024-03-25T18:35:40.979 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.5 M 2024-03-25T18:35:40.979 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:41.101 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 571 kB/s | 69 kB 00:00 2024-03-25T18:35:41.192 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:41.192 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:35:41.514 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:35:41.514 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:41.514 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:41.514 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:35:41.514 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:41.514 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:41.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2024-03-25T18:35:42.382 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:42.534 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:42.534 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:42.534 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-25T18:35:42.534 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-25T18:35:42.535 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:42.536 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 332 k 2024-03-25T18:35:42.536 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.5 M 2024-03-25T18:35:42.536 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:35:42.550 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 168 kB/s | 264 kB 00:01 2024-03-25T18:35:42.551 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:42.552 INFO:teuthology.orchestra.run.smithi193.stdout:Total 211 kB/s | 332 kB 00:01 2024-03-25T18:35:42.552 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:42.562 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:42.562 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:42.601 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:42.601 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:42.669 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 517 kB/s | 69 kB 00:00 2024-03-25T18:35:42.771 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:42.966 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:35:43.011 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:43.026 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:43.026 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:43.026 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:35:43.027 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:43.765 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:35:44.126 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:35:44.126 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:44.126 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:44.127 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:44.127 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:35:44.127 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:44.127 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:44.194 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 159 kB/s | 264 kB 00:01 2024-03-25T18:35:44.195 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:44.196 INFO:teuthology.orchestra.run.smithi071.stdout:Total 200 kB/s | 332 kB 00:01 2024-03-25T18:35:44.196 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:44.206 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:44.206 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:44.247 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:44.247 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:44.400 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:44.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados-devel 2024-03-25T18:35:44.593 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:35:44.621 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:44.637 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:44.637 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:44.638 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:35:44.638 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:45.032 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:45.180 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:45.181 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:35:45.181 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:45.181 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:35:45.181 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:45.182 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:45.182 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:45.182 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:35:45.182 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:45.182 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 127 k 2024-03-25T18:35:45.182 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 456 k 2024-03-25T18:35:45.182 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:45.421 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:35:45.748 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:35:45.748 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:45.748 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:45.748 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:45.748 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:35:45.748 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:45.748 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:46.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2024-03-25T18:35:46.256 INFO:teuthology.orchestra.run.smithi193.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 118 kB/s | 127 kB 00:01 2024-03-25T18:35:46.257 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:46.257 INFO:teuthology.orchestra.run.smithi193.stdout:Total 118 kB/s | 127 kB 00:01 2024-03-25T18:35:46.257 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:46.264 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:46.264 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:46.324 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:46.324 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:46.474 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:46.589 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:46.656 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:05 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:46.810 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:46.810 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:46.810 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T18:35:46.810 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:46.810 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:46.811 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:35:46.811 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:46.811 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:46.811 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:46.811 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:35:46.811 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:46.811 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 127 k 2024-03-25T18:35:46.811 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 456 k 2024-03-25T18:35:46.812 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:35:47.043 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:47.359 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:47.359 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:47.359 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:47.509 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 182 kB/s | 127 kB 00:00 2024-03-25T18:35:47.509 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:47.510 INFO:teuthology.orchestra.run.smithi071.stdout:Total 181 kB/s | 127 kB 00:00 2024-03-25T18:35:47.510 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:47.516 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:47.516 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:47.573 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:47.574 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:47.626 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs2 2024-03-25T18:35:47.728 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:47.833 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:48.202 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:48.302 INFO:teuthology.orchestra.run.smithi193.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:48.311 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:48.356 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:48.358 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:35:48.358 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:48.427 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs-devel 2024-03-25T18:35:48.584 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:48.584 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:48.584 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:48.584 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:48.584 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:48.584 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:48.827 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2024-03-25T18:35:48.991 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:49.140 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:49.140 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:49.140 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:35:49.140 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:49.140 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:49.140 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:35:49.141 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:49.141 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:49.141 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:49.141 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:35:49.141 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:49.141 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 31 k 2024-03-25T18:35:49.141 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 140 k 2024-03-25T18:35:49.141 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:49.401 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:49.501 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:49.557 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:49.558 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:35:49.558 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:49.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2024-03-25T18:35:49.702 INFO:teuthology.orchestra.run.smithi193.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 56 kB/s | 31 kB 00:00 2024-03-25T18:35:49.703 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:49.703 INFO:teuthology.orchestra.run.smithi193.stdout:Total 55 kB/s | 31 kB 00:00 2024-03-25T18:35:49.703 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:49.707 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:49.707 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:49.732 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:49.733 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:49.843 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:50.195 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:50.322 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:50.346 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:50.347 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:35:50.348 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:50.348 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 31 k 2024-03-25T18:35:50.348 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 140 k 2024-03-25T18:35:50.348 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:35:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:50.617 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:50.617 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:50.617 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:50.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados2 2024-03-25T18:35:50.979 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 49 kB/s | 31 kB 00:00 2024-03-25T18:35:50.979 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:50.980 INFO:teuthology.orchestra.run.smithi071.stdout:Total 49 kB/s | 31 kB 00:00 2024-03-25T18:35:50.980 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:50.984 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:50.984 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:51.014 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:51.140 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:51.237 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:51.345 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:51.453 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:51.505 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:51.507 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:35:51.507 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:51.557 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librbd1 2024-03-25T18:35:51.653 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:51.917 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:51.917 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:51.917 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:51.917 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:51.917 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:51.917 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:52.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2024-03-25T18:35:52.118 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:52.215 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:52.268 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:52.269 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:35:52.269 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:52.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rados 2024-03-25T18:35:52.652 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:52.761 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:52.814 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:52.815 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:35:52.816 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:52.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2024-03-25T18:35:52.882 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:52.979 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:53.030 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:53.032 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:35:53.032 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:53.087 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rgw 2024-03-25T18:35:53.418 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:53.515 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:53.566 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:53.566 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:35:53.567 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:53.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2024-03-25T18:35:53.658 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:53.764 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:53.821 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:53.822 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:35:53.822 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:53.887 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-cephfs 2024-03-25T18:35:54.179 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:54.276 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:54.330 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:54.331 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:35:54.331 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:54.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2024-03-25T18:35:54.474 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:54.575 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:54.630 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:54.631 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:35:54.631 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:54.691 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rbd 2024-03-25T18:35:54.941 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:55.042 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:55.095 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:55.096 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:35:55.096 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:55.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2024-03-25T18:35:55.251 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:55.347 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:55.402 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:55.402 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T18:35:55.402 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:55.449 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-fuse 2024-03-25T18:35:55.725 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:55.823 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:55.876 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:35:55.876 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:55.920 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2024-03-25T18:35:56.014 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:56.162 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:56.163 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:56.163 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:56.163 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:56.163 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:56.163 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:35:56.164 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:56.164 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:56.164 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:56.164 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:35:56.164 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:56.164 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 85 k 2024-03-25T18:35:56.164 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 231 k 2024-03-25T18:35:56.164 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:56.479 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:56.575 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:56.626 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:56.627 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T18:35:56.627 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:56.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2024-03-25T18:35:56.989 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 103 kB/s | 85 kB 00:00 2024-03-25T18:35:56.989 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:56.990 INFO:teuthology.orchestra.run.smithi193.stdout:Total 103 kB/s | 85 kB 00:00 2024-03-25T18:35:56.990 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:35:56.998 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:35:56.999 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:35:57.059 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:35:57.059 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:35:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:35:57.246 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:57.291 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:57.394 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:57.395 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:57.395 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:57.395 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:57.395 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:57.395 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:35:57.395 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:57.396 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:57.396 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:57.396 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:35:57.396 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:57.396 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 85 k 2024-03-25T18:35:57.396 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 231 k 2024-03-25T18:35:57.396 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:35:57.716 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:57.923 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 162 kB/s | 85 kB 00:00 2024-03-25T18:35:57.923 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:57.924 INFO:teuthology.orchestra.run.smithi071.stdout:Total 161 kB/s | 85 kB 00:00 2024-03-25T18:35:57.924 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:35:57.931 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:35:57.932 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:35:57.988 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:35:57.989 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:35:58.017 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:58.017 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:58.017 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:35:58.017 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:58.018 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:58.018 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:35:58.131 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:35:58.230 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:58.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-mirror 2024-03-25T18:35:58.684 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:58.814 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:35:58.967 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:35:58.968 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:58.968 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:35:58.968 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:58.968 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:35:58.968 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:35:58.968 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:58.968 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:35:58.969 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:35:58.969 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:35:58.969 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:35:58.969 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 3.1 M 2024-03-25T18:35:58.969 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 13 M 2024-03-25T18:35:58.969 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:35:58.973 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:58.973 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:58.973 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:35:58.973 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:58.973 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:58.974 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:35:59.256 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2024-03-25T18:35:59.827 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:35:59.982 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.1 M 2024-03-25T18:35:59.983 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 13 M 2024-03-25T18:35:59.984 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:36:05.201 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 507 kB/s | 3.1 MB 00:06 2024-03-25T18:36:05.201 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:05.201 INFO:teuthology.orchestra.run.smithi193.stdout:Total 507 kB/s | 3.1 MB 00:06 2024-03-25T18:36:05.202 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:36:05.213 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:36:05.213 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:36:05.290 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:36:05.291 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:36:05.730 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:36:05.792 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:05.817 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:05.817 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:36:05.817 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:36:05.817 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:36:05.817 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:36:05.817 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:06.658 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 474 kB/s | 3.1 MB 00:06 2024-03-25T18:36:06.659 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:06.659 INFO:teuthology.orchestra.run.smithi071.stdout:Total 474 kB/s | 3.1 MB 00:06 2024-03-25T18:36:06.659 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:36:06.670 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:36:06.670 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:36:06.746 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:36:06.747 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:36:06.923 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:06.923 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:06.923 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:36:06.923 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:06.923 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:06.923 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:36:07.168 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:36:07.187 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-nbd 2024-03-25T18:36:07.225 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:07.249 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:07.249 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:36:07.249 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:36:07.249 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:36:07.249 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:36:07.249 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:07.762 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:28 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:36:07.912 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:36:07.913 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:36:07.913 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-25T18:36:07.913 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:36:07.913 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 171 k 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 494 k 2024-03-25T18:36:07.914 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:36:08.366 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:08.366 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:08.366 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:36:08.366 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:08.366 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:08.366 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:36:08.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2024-03-25T18:36:08.663 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 229 kB/s | 171 kB 00:00 2024-03-25T18:36:08.663 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:08.664 INFO:teuthology.orchestra.run.smithi193.stdout:Total 229 kB/s | 171 kB 00:00 2024-03-25T18:36:08.664 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:36:08.671 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:36:08.671 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:36:08.728 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:36:08.729 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:36:08.877 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:36:08.979 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:09.213 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:28 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:36:09.361 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:36:09.362 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 171 k 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 494 k 2024-03-25T18:36:09.363 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:36:09.408 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:09.730 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:09.731 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:09.731 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:36:09.731 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:09.731 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:09.731 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:36:09.939 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:36:10.444 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 159 kB/s | 171 kB 00:01 2024-03-25T18:36:10.445 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:10.445 INFO:teuthology.orchestra.run.smithi071.stdout:Total 158 kB/s | 171 kB 00:01 2024-03-25T18:36:10.445 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:36:10.453 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:36:10.454 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:36:10.510 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:36:10.511 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:36:10.511 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:36:10.658 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:36:10.663 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-03-25T18:36:10.664 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:36:10.665 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 91 M 2024-03-25T18:36:10.665 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 498 M 2024-03-25T18:36:10.665 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:36:10.760 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:11.178 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:11.462 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:11.462 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:11.462 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:36:11.462 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:11.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:11.463 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:36:11.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:36:12.274 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:36:12.425 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-25T18:36:12.426 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:36:12.427 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 91 M 2024-03-25T18:36:12.427 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 498 M 2024-03-25T18:36:12.427 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:37:10.677 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 395 kB/s | 22 MB 00:58 2024-03-25T18:37:19.608 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 333 kB/s | 22 MB 01:08 2024-03-25T18:39:02.177 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 407 kB/s | 68 MB 02:51 2024-03-25T18:39:02.178 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:39:02.178 INFO:teuthology.orchestra.run.smithi193.stdout:Total 541 kB/s | 91 MB 02:51 2024-03-25T18:39:02.178 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:39:02.182 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:39:02.182 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:39:02.770 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:39:02.770 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:39:05.884 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:39:12.957 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:39:13.102 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:16.414 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:16.415 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:39:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:39:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:39:16.672 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:39:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:16.672 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:39:16.673 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:39:16.877 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:39:17.456 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:08:38 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:39:17.616 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:39:17.617 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:39:17.617 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:39:17.617 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 416 M 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.7 G 2024-03-25T18:39:17.618 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:39:32.008 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 350 kB/s | 68 MB 03:19 2024-03-25T18:39:32.009 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:39:32.009 INFO:teuthology.orchestra.run.smithi071.stdout:Total 465 kB/s | 91 MB 03:19 2024-03-25T18:39:32.009 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:39:32.013 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:39:32.013 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:39:32.606 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:39:32.606 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:39:35.706 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:39:42.888 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:39:43.032 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:46.489 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:46.489 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:39:46.756 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:39:46.757 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:39:46.757 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:39:46.757 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:46.757 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:46.757 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:39:46.757 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:39:46.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:39:47.567 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:09:06 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:39:47.723 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:39:47.723 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:39:47.723 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T18:39:47.723 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:39:47.724 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:39:47.724 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:39:47.724 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:39:47.724 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:39:47.724 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:39:47.724 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:39:47.724 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:39:47.725 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 416 M 2024-03-25T18:39:47.725 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.7 G 2024-03-25T18:39:47.725 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:53:37.224 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 514 kB/s | 416 MB 13:49 2024-03-25T18:53:37.224 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:53:37.225 INFO:teuthology.orchestra.run.smithi071.stdout:Total 514 kB/s | 416 MB 13:49 2024-03-25T18:53:37.225 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:53:37.228 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:53:37.228 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:53:39.703 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:53:39.703 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:54:21.958 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:54:22.048 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:54:27.002 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:54:27.256 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:54:27.256 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:54:27.256 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:54:27.256 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:54:27.257 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:54:27.257 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:54:27.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:54:28.075 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:23:47 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:54:28.224 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:54:28.225 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:54:28.225 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T18:54:28.225 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.4 M 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 6.4 M 2024-03-25T18:54:28.226 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:54:30.868 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 553 kB/s | 1.4 MB 00:02 2024-03-25T18:54:30.868 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:54:30.869 INFO:teuthology.orchestra.run.smithi071.stdout:Total 553 kB/s | 1.4 MB 00:02 2024-03-25T18:54:30.869 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T18:54:30.873 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T18:54:30.873 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T18:54:30.918 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T18:54:30.918 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T18:54:31.178 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T18:54:31.266 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:54:31.706 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:54:31.973 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:54:31.973 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:54:31.973 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T18:54:31.973 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:54:31.973 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:54:31.973 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T18:54:32.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:54:32.732 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:23:51 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T18:54:32.879 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T18:54:32.880 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:54:32.880 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T18:54:32.880 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:54:32.880 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T18:54:32.880 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:54:32.880 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:54:32.881 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T18:54:32.881 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T18:54:32.881 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T18:54:32.881 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T18:54:32.881 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 176 M 2024-03-25T18:54:32.881 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 753 M 2024-03-25T18:54:32.881 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T18:55:40.147 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 434 kB/s | 416 MB 16:22 2024-03-25T18:55:40.148 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:55:40.148 INFO:teuthology.orchestra.run.smithi193.stdout:Total 434 kB/s | 416 MB 16:22 2024-03-25T18:55:40.148 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:55:40.152 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:55:40.152 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:55:42.619 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:55:42.620 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:56:25.062 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:56:25.166 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:56:30.124 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:56:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:56:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:56:30.389 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:56:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:56:30.389 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:56:30.622 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:56:31.209 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:25:52 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:56:31.369 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:56:31.370 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:56:31.371 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 1.4 M 2024-03-25T18:56:31.371 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 6.4 M 2024-03-25T18:56:31.371 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:56:36.836 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 267 kB/s | 1.4 MB 00:05 2024-03-25T18:56:36.837 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:36.837 INFO:teuthology.orchestra.run.smithi193.stdout:Total 267 kB/s | 1.4 MB 00:05 2024-03-25T18:56:36.838 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:56:36.842 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:56:36.842 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:56:36.888 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:56:36.888 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:56:37.146 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:56:37.243 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:56:37.659 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:56:37.932 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:56:37.932 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:56:37.932 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:56:37.932 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:37.932 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:56:37.932 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:56:38.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:56:38.661 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:25:59 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T18:56:38.809 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:56:38.810 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:56:38.810 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 176 M 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 753 M 2024-03-25T18:56:38.811 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:01:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 431 kB/s | 176 MB 06:56 2024-03-25T19:01:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:01:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:Total 431 kB/s | 176 MB 06:56 2024-03-25T19:01:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:01:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:01:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:01:30.868 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:01:30.868 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:01:48.871 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:01:48.977 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:01:53.515 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:01:53.773 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:01:53.773 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:01:53.774 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:01:53.774 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:01:53.774 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:01:53.774 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:01:53.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:01:54.554 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:31:13 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:01:54.701 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:01:54.702 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:01:54.702 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:01:54.702 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 672 M 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 3.1 G 2024-03-25T19:01:54.703 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:04:08.475 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 400 kB/s | 176 MB 07:29 2024-03-25T19:04:08.476 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:08.476 INFO:teuthology.orchestra.run.smithi193.stdout:Total 400 kB/s | 176 MB 07:29 2024-03-25T19:04:08.476 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:04:08.479 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:04:08.479 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:04:09.539 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:04:09.540 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:04:27.757 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:04:27.920 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:04:32.532 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:04:32.806 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:04:32.806 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:04:32.806 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:04:32.806 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:32.806 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:04:32.806 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:04:32.993 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:04:33.560 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:33:54 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:04:33.710 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:04:33.710 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:04:33.710 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:04:33.710 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:04:33.710 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:04:33.711 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:04:33.711 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:04:33.711 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:04:33.711 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:04:33.711 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:04:33.711 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:04:33.711 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 672 M 2024-03-25T19:04:33.712 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 3.1 G 2024-03-25T19:04:33.712 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:28:03.795 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 439 kB/s | 672 MB 26:09 2024-03-25T19:28:03.795 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:03.796 INFO:teuthology.orchestra.run.smithi071.stdout:Total 439 kB/s | 672 MB 26:09 2024-03-25T19:28:03.796 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:28:03.799 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:28:03.800 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:28:07.762 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:28:07.763 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:28:31.186 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 479 kB/s | 672 MB 23:57 2024-03-25T19:28:31.187 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:31.187 INFO:teuthology.orchestra.run.smithi193.stdout:Total 479 kB/s | 672 MB 23:57 2024-03-25T19:28:31.188 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:28:31.192 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:28:31.192 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:28:35.143 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:28:35.144 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:29:18.489 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:29:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:28.601 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:28.865 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:28.865 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:29:28.865 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:29:28.865 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:28.865 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:29:28.865 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:29:29.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:29:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:58:48 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:29:29.815 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:29:29.867 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:29:29.868 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-25T19:29:29.868 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:29:29.947 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:29:30.506 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:58:49 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:29:30.655 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:29:30.656 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 20 M 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 103 M 2024-03-25T19:29:30.657 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:29:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:29:45.987 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:49.204 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 20 MB 00:18 2024-03-25T19:29:49.205 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:49.206 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.1 MB/s | 20 MB 00:18 2024-03-25T19:29:49.206 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:29:49.210 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:29:49.210 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:29:49.360 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:29:49.360 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:29:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:29:51.861 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:52.811 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:53.068 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:53.068 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:29:53.068 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:29:53.069 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:53.069 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:29:53.069 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:29:53.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:29:53.817 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:59:12 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:29:53.965 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:29:53.966 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:29:53.966 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:29:53.966 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:29:53.966 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:29:53.966 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:29:53.966 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:29:53.967 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:29:53.967 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:29:53.967 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:29:53.967 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:29:53.967 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 48 M 2024-03-25T19:29:53.967 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 173 M 2024-03-25T19:29:53.967 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:29:55.488 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:55.765 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:29:55.766 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:29:55.766 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:29:55.766 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:55.766 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:29:55.766 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:29:56.053 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:29:56.616 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:59:17 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:29:56.715 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:29:56.767 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:29:56.768 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T19:29:56.768 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:29:56.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:29:57.402 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:59:18 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:29:57.566 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:29:57.566 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:29:57.566 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 20 M 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 103 M 2024-03-25T19:29:57.567 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:30:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.5 MB/s | 20 MB 00:13 2024-03-25T19:30:10.664 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:10.665 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.5 MB/s | 20 MB 00:13 2024-03-25T19:30:10.665 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:30:10.669 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:30:10.669 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:30:10.818 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:30:10.819 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:30:13.039 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:30:13.145 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:14.116 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:14.380 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:14.381 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:30:14.381 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:30:14.381 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:30:14.381 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:30:14.381 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:30:14.552 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:30:15.128 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:59:36 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:30:15.276 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:30:15.276 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:30:15.277 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:30:15.278 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 48 M 2024-03-25T19:30:15.278 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 173 M 2024-03-25T19:30:15.278 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:30:49.647 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 878 kB/s | 48 MB 00:55 2024-03-25T19:30:49.648 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:49.648 INFO:teuthology.orchestra.run.smithi071.stdout:Total 878 kB/s | 48 MB 00:55 2024-03-25T19:30:49.648 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:30:49.652 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:30:49.653 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:30:49.966 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:30:49.966 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:30:54.723 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:30:54.802 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:56.119 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:56.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:56.370 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:30:56.370 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:30:56.370 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:30:56.370 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:30:56.370 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:30:56.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:30:57.118 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:00:16 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:30:57.266 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:30:57.267 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:30:57.267 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:30:57.267 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:30:57.267 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:30:57.267 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:30:57.267 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:30:57.267 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:30:57.268 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:30:57.268 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:30:57.268 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:30:57.268 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 99 M 2024-03-25T19:30:57.268 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 449 M 2024-03-25T19:30:57.268 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:31:11.734 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 866 kB/s | 48 MB 00:56 2024-03-25T19:31:11.734 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:11.735 INFO:teuthology.orchestra.run.smithi193.stdout:Total 866 kB/s | 48 MB 00:56 2024-03-25T19:31:11.735 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:31:11.739 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:31:11.739 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:31:12.053 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:31:12.053 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:31:16.780 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:31:16.924 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:18.237 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:18.510 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:18.511 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:31:18.511 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:31:18.511 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:18.511 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:31:18.511 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:31:18.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:31:19.252 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:00:40 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:31:19.403 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:31:19.404 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:31:19.404 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:31:19.404 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:31:19.405 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:31:19.405 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:31:19.405 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:31:19.405 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:31:19.405 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:31:19.405 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:31:19.406 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:31:19.406 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 99 M 2024-03-25T19:31:19.406 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 449 M 2024-03-25T19:31:19.406 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:32:39.087 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 991 kB/s | 99 MB 01:41 2024-03-25T19:32:39.088 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:39.088 INFO:teuthology.orchestra.run.smithi071.stdout:Total 991 kB/s | 99 MB 01:41 2024-03-25T19:32:39.088 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:32:39.091 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:32:39.091 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:32:39.700 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:32:39.701 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:32:49.963 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:32:50.067 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:53.072 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:53.330 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:53.330 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:32:53.330 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:32:53.330 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:53.330 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:32:53.330 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:32:53.516 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:32:54.077 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:02:13 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:32:54.225 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 287 M 2024-03-25T19:32:54.226 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.2 G 2024-03-25T19:32:54.227 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:33:12.278 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 894 kB/s | 99 MB 01:52 2024-03-25T19:33:12.279 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:12.279 INFO:teuthology.orchestra.run.smithi193.stdout:Total 894 kB/s | 99 MB 01:52 2024-03-25T19:33:12.279 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:33:12.283 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:33:12.283 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:33:12.893 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:33:12.893 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:33:23.123 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:33:23.225 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:26.135 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:26.404 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:26.404 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:33:26.404 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:33:26.405 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:26.405 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:33:26.405 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:33:26.583 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:33:27.147 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:02:48 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:33:27.295 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:33:27.295 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:33:27.295 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:33:27.295 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:33:27.295 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:33:27.296 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:33:27.296 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:33:27.296 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:33:27.296 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:33:27.296 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:33:27.296 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:33:27.296 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 287 M 2024-03-25T19:33:27.297 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.2 G 2024-03-25T19:33:27.297 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:37:07.358 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 287 MB 03:40 2024-03-25T19:37:07.359 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:07.359 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.3 MB/s | 287 MB 03:40 2024-03-25T19:37:07.359 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:37:07.362 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:37:07.363 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:37:09.073 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:37:09.073 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:37:11.237 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 287 MB 04:17 2024-03-25T19:37:11.237 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:11.238 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.1 MB/s | 287 MB 04:17 2024-03-25T19:37:11.238 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:37:11.242 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:37:11.242 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:37:12.953 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:37:12.954 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:37:38.656 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:37:38.843 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:42.459 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:37:42.564 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:46.217 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:46.531 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:46.531 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:46.531 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:37:46.531 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:46.531 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:46.532 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:37:46.714 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:37:47.288 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:07:08 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:37:47.447 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:37:47.447 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:47.447 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:37:47.447 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:47.447 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:37:47.447 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:37:47.447 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:47.448 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:37:47.448 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:47.448 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:37:47.448 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:47.448 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 5.9 M 2024-03-25T19:37:47.448 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 28 M 2024-03-25T19:37:47.448 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:37:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:50.493 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:50.494 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:50.494 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:37:50.494 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:50.494 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:50.494 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:37:50.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:37:51.307 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:07:10 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:37:51.455 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:37:51.456 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:37:51.456 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:37:51.456 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:37:51.456 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:37:51.456 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:37:51.457 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:51.457 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:37:51.457 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:37:51.457 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:37:51.457 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:51.457 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 5.9 M 2024-03-25T19:37:51.457 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 28 M 2024-03-25T19:37:51.457 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:37:51.980 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.3 MB/s | 5.9 MB 00:04 2024-03-25T19:37:51.981 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:51.982 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.3 MB/s | 5.9 MB 00:04 2024-03-25T19:37:51.982 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:37:51.986 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:37:51.986 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:37:52.059 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:37:52.059 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:37:52.797 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:37:52.913 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:53.467 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:53.743 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:53.743 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:53.743 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:37:53.743 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:53.743 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:53.743 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:37:53.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:37:54.512 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:07:15 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:37:54.662 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:54.663 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:37:54.664 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:54.664 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 109 k 2024-03-25T19:37:54.664 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 440 k 2024-03-25T19:37:54.664 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:37:55.333 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.5 MB/s | 5.9 MB 00:03 2024-03-25T19:37:55.334 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:55.334 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.5 MB/s | 5.9 MB 00:03 2024-03-25T19:37:55.334 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:37:55.338 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:37:55.338 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:37:55.358 INFO:teuthology.orchestra.run.smithi193.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 156 kB/s | 109 kB 00:00 2024-03-25T19:37:55.359 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:55.359 INFO:teuthology.orchestra.run.smithi193.stdout:Total 156 kB/s | 109 kB 00:00 2024-03-25T19:37:55.359 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:37:55.363 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:37:55.363 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:37:55.396 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:37:55.397 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:37:55.410 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:37:55.410 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:37:55.509 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:37:55.615 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:55.983 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:56.146 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:37:56.256 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:56.259 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:56.259 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:56.259 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:37:56.259 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:56.259 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:56.259 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:37:56.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:37:56.839 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:56.994 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:07:17 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:37:57.090 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:37:57.091 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:57.091 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:37:57.091 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:57.091 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:57.091 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:37:57.141 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:37:57.142 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:37:57.143 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:37:57.143 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 5.4 M 2024-03-25T19:37:57.143 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 25 M 2024-03-25T19:37:57.143 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:37:57.287 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:37:57.843 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:07:16 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:37:57.993 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:37:57.994 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:37:57.995 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:57.995 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 109 k 2024-03-25T19:37:57.995 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 440 k 2024-03-25T19:37:57.995 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:37:58.560 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 192 kB/s | 109 kB 00:00 2024-03-25T19:37:58.561 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:58.561 INFO:teuthology.orchestra.run.smithi071.stdout:Total 192 kB/s | 109 kB 00:00 2024-03-25T19:37:58.561 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:37:58.565 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:37:58.565 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:37:58.599 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:37:58.599 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:37:58.713 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:37:58.817 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:59.234 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:59.489 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:59.489 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:59.490 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:37:59.490 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:59.490 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:37:59.490 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:37:59.674 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:38:00.231 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:07:19 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:38:00.379 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:38:00.380 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:00.380 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:38:00.380 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:00.380 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:38:00.380 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:38:00.381 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:00.381 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:38:00.381 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:00.381 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:38:00.381 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:00.381 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 5.4 M 2024-03-25T19:38:00.381 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 25 M 2024-03-25T19:38:00.381 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:38:00.719 INFO:teuthology.orchestra.run.smithi193.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 5.4 MB 00:03 2024-03-25T19:38:00.720 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:00.721 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.5 MB/s | 5.4 MB 00:03 2024-03-25T19:38:00.721 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:38:00.725 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:38:00.725 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:38:00.794 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:38:00.794 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:38:01.458 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:38:01.559 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:02.088 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:02.364 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:02.364 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:02.364 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:38:02.364 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:02.364 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:02.364 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:38:02.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados2-debuginfo 2024-03-25T19:38:03.107 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:07:24 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:38:03.258 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:38:03.259 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:03.259 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:38:03.259 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:03.259 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:38:03.259 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:38:03.260 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:03.260 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:38:03.260 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:03.260 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:38:03.260 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:03.260 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 44 M 2024-03-25T19:38:03.260 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 253 M 2024-03-25T19:38:03.260 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:38:03.530 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 5.4 MB 00:03 2024-03-25T19:38:03.530 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:03.531 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.7 MB/s | 5.4 MB 00:03 2024-03-25T19:38:03.531 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:38:03.535 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:38:03.535 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:38:03.603 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:38:03.604 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:38:04.266 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:38:04.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:04.868 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:05.130 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:05.131 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:05.131 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:38:05.131 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:05.131 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:05.131 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:38:05.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2-debuginfo 2024-03-25T19:38:05.894 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:07:24 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:38:06.042 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:38:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:06.044 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 44 M 2024-03-25T19:38:06.044 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 253 M 2024-03-25T19:38:06.044 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:38:10.522 INFO:teuthology.orchestra.run.smithi193.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 6.1 MB/s | 44 MB 00:07 2024-03-25T19:38:10.523 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:10.523 INFO:teuthology.orchestra.run.smithi193.stdout:Total 6.1 MB/s | 44 MB 00:07 2024-03-25T19:38:10.524 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:38:10.528 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:38:10.528 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:38:10.827 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:38:10.827 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:38:15.897 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:38:15.997 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:18.081 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:18.081 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:18.081 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:38:18.081 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:18.081 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:18.081 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:38:18.289 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:38:18.854 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:07:39 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:38:19.003 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:38:19.004 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:38:19.004 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:19.004 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:38:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:38:19.004 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:19.004 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:38:19.005 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:19.005 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:38:19.005 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:19.005 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 52 M 2024-03-25T19:38:19.005 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 308 M 2024-03-25T19:38:19.005 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:38:21.796 INFO:teuthology.orchestra.run.smithi071.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 2.8 MB/s | 44 MB 00:15 2024-03-25T19:38:21.797 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:21.797 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.8 MB/s | 44 MB 00:15 2024-03-25T19:38:21.798 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:38:21.801 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:38:21.802 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:38:22.102 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:38:22.102 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:38:27.145 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:38:27.244 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:29.091 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:29.360 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:29.360 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:29.361 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:38:29.361 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:29.361 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:29.361 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:38:29.578 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:38:30.142 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:07:49 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:38:30.290 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:38:30.291 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:30.291 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:38:30.291 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:30.291 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:38:30.291 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:38:30.291 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:30.291 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:38:30.291 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:38:30.292 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:38:30.292 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:38:30.292 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 52 M 2024-03-25T19:38:30.292 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 308 M 2024-03-25T19:38:30.292 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:38:45.130 INFO:teuthology.orchestra.run.smithi193.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.0 MB/s | 52 MB 00:26 2024-03-25T19:38:45.130 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:45.131 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.0 MB/s | 52 MB 00:26 2024-03-25T19:38:45.131 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:38:45.135 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:38:45.135 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:38:45.481 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:38:45.482 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:38:51.452 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:38:51.845 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:53.959 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:54.248 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:54.248 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:54.248 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:38:54.248 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:54.248 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:54.249 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:38:54.444 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:38:55.002 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:08:16 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:38:55.152 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:38:55.153 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:55.154 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 615 k 2024-03-25T19:38:55.154 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.1 M 2024-03-25T19:38:55.154 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:38:57.224 INFO:teuthology.orchestra.run.smithi193.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 297 kB/s | 615 kB 00:02 2024-03-25T19:38:57.225 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:57.225 INFO:teuthology.orchestra.run.smithi193.stdout:Total 297 kB/s | 615 kB 00:02 2024-03-25T19:38:57.225 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:38:57.229 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:38:57.229 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:38:57.270 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:38:57.271 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:38:57.436 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:38:57.539 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:57.929 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:58.199 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:58.199 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:58.199 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:38:58.199 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:58.200 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:58.200 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:38:58.400 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:38:58.962 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:08:19 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:38:59.114 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:38:59.115 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:59.115 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:38:59.115 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:59.115 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:38:59.115 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:38:59.115 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:59.115 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:38:59.116 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:38:59.116 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:38:59.116 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:38:59.116 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 771 k 2024-03-25T19:38:59.116 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.7 M 2024-03-25T19:38:59.116 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:39:00.052 INFO:teuthology.orchestra.run.smithi193.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 823 kB/s | 771 kB 00:00 2024-03-25T19:39:00.052 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:00.053 INFO:teuthology.orchestra.run.smithi193.stdout:Total 822 kB/s | 771 kB 00:00 2024-03-25T19:39:00.053 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:39:00.056 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:39:00.056 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:39:00.096 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:39:00.096 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:39:00.274 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:39:00.376 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:00.551 INFO:teuthology.orchestra.run.smithi071.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 52 MB 00:30 2024-03-25T19:39:00.552 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:00.552 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.7 MB/s | 52 MB 00:30 2024-03-25T19:39:00.552 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:39:00.556 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:39:00.556 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:39:00.767 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:00.902 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:39:00.903 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:39:01.038 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:01.038 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:01.038 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:39:01.038 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:01.039 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:01.039 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:39:01.242 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:39:01.803 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:08:22 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:39:01.951 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:39:01.952 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:01.953 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 1.1 M 2024-03-25T19:39:01.953 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 4.3 M 2024-03-25T19:39:01.953 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:39:02.939 INFO:teuthology.orchestra.run.smithi193.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 1.1 MB/s | 1.1 MB 00:00 2024-03-25T19:39:02.939 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:02.939 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.1 MB/s | 1.1 MB 00:00 2024-03-25T19:39:02.940 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:39:02.943 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:39:02.943 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:39:02.985 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:39:02.985 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:39:03.203 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:39:03.304 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:03.666 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:03.947 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:03.947 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:03.947 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:39:03.947 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:03.948 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:03.948 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:39:04.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:39:04.701 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:08:25 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:39:04.851 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:04.852 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:39:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:04.853 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 337 k 2024-03-25T19:39:04.853 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.0 M 2024-03-25T19:39:04.853 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:39:05.275 INFO:teuthology.orchestra.run.smithi193.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 798 kB/s | 337 kB 00:00 2024-03-25T19:39:05.276 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:05.276 INFO:teuthology.orchestra.run.smithi193.stdout:Total 795 kB/s | 337 kB 00:00 2024-03-25T19:39:05.276 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:39:05.280 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:39:05.280 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:39:05.314 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:39:05.315 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:39:05.454 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:39:05.558 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:05.935 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:06.212 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:06.212 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:06.213 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:39:06.213 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:06.213 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:06.213 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:39:06.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:39:06.877 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:39:06.957 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:08:27 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:39:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:07.106 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:39:07.107 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:07.108 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 509 k 2024-03-25T19:39:07.108 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.5 M 2024-03-25T19:39:07.108 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:39:07.814 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 721 kB/s | 509 kB 00:00 2024-03-25T19:39:07.815 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:07.815 INFO:teuthology.orchestra.run.smithi193.stdout:Total 719 kB/s | 509 kB 00:00 2024-03-25T19:39:07.815 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:39:07.819 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:39:07.819 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:39:07.857 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:39:07.857 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:39:08.020 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:39:08.130 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:08.494 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:08.772 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:08.772 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:08.772 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:39:08.772 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:08.772 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:08.772 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:39:08.978 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:39:09.324 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:09.543 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:08:30 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:39:09.586 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:09.586 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:09.586 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:39:09.586 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:09.586 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:09.586 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:39:09.694 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:39:09.694 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:39:09.695 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:09.696 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 52 M 2024-03-25T19:39:09.696 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 298 M 2024-03-25T19:39:09.696 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:39:09.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:39:10.334 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:08:29 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:39:10.484 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:39:10.484 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:39:10.484 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:10.484 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:39:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:39:10.484 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:10.484 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:39:10.485 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:10.485 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:39:10.485 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:10.485 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 615 k 2024-03-25T19:39:10.485 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.1 M 2024-03-25T19:39:10.485 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:39:11.661 INFO:teuthology.orchestra.run.smithi071.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 523 kB/s | 615 kB 00:01 2024-03-25T19:39:11.661 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:11.662 INFO:teuthology.orchestra.run.smithi071.stdout:Total 522 kB/s | 615 kB 00:01 2024-03-25T19:39:11.662 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:39:11.666 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:39:11.666 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:39:11.706 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:39:11.707 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:39:11.875 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:39:11.984 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:12.443 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:12.700 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:39:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:12.700 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:12.700 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:39:12.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:39:13.445 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:08:32 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:39:13.595 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:13.596 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:39:13.597 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:13.597 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 771 k 2024-03-25T19:39:13.597 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.7 M 2024-03-25T19:39:13.597 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:39:14.547 INFO:teuthology.orchestra.run.smithi071.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 812 kB/s | 771 kB 00:00 2024-03-25T19:39:14.548 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:14.548 INFO:teuthology.orchestra.run.smithi071.stdout:Total 810 kB/s | 771 kB 00:00 2024-03-25T19:39:14.548 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:39:14.552 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:39:14.552 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:39:14.591 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:39:14.592 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:39:14.771 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:39:14.870 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:15.269 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:15.524 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:15.525 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:15.525 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:39:15.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:15.525 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:15.525 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:39:15.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:39:16.299 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:08:35 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:39:16.447 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.1 M 2024-03-25T19:39:16.448 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 4.3 M 2024-03-25T19:39:16.449 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:39:17.791 INFO:teuthology.orchestra.run.smithi071.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 857 kB/s | 1.1 MB 00:01 2024-03-25T19:39:17.792 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:17.793 INFO:teuthology.orchestra.run.smithi071.stdout:Total 856 kB/s | 1.1 MB 00:01 2024-03-25T19:39:17.793 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:39:17.797 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:39:17.797 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:39:17.840 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:39:17.840 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:39:18.068 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:39:18.173 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:18.831 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:39:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:18.831 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:39:19.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:39:19.604 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:08:38 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:39:19.755 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:39:19.756 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:19.757 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 337 k 2024-03-25T19:39:19.757 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.0 M 2024-03-25T19:39:19.757 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:39:20.411 INFO:teuthology.orchestra.run.smithi071.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 516 kB/s | 337 kB 00:00 2024-03-25T19:39:20.411 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:20.412 INFO:teuthology.orchestra.run.smithi071.stdout:Total 515 kB/s | 337 kB 00:00 2024-03-25T19:39:20.412 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:39:20.415 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:39:20.415 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:39:20.451 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:39:20.452 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:39:20.585 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:39:20.686 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:21.067 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:21.324 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:21.324 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:21.324 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:39:21.324 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:21.325 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:21.325 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:39:21.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:39:22.123 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:08:41 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:39:22.272 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:39:22.273 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:22.273 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:39:22.273 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:22.273 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:39:22.273 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:39:22.273 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:22.274 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:39:22.274 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:22.274 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:39:22.274 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:22.274 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 509 k 2024-03-25T19:39:22.274 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.5 M 2024-03-25T19:39:22.274 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:39:23.047 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 658 kB/s | 509 kB 00:00 2024-03-25T19:39:23.048 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:23.048 INFO:teuthology.orchestra.run.smithi071.stdout:Total 657 kB/s | 509 kB 00:00 2024-03-25T19:39:23.048 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:39:23.052 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:39:23.052 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:39:23.090 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:39:23.091 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:39:23.250 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:39:23.346 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:23.713 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:39:24.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:39:24.490 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 52 MB 00:14 2024-03-25T19:39:24.491 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:24.491 INFO:teuthology.orchestra.run.smithi193.stdout:Total 3.5 MB/s | 52 MB 00:14 2024-03-25T19:39:24.492 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:39:24.496 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:39:24.496 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:39:24.739 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:08:43 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:39:24.830 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:39:24.831 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:39:24.887 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:39:24.888 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:24.889 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 52 M 2024-03-25T19:39:24.889 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 298 M 2024-03-25T19:39:24.889 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:39:29.074 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 12 MB/s | 52 MB 00:04 2024-03-25T19:39:29.075 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:29.075 INFO:teuthology.orchestra.run.smithi071.stdout:Total 12 MB/s | 52 MB 00:04 2024-03-25T19:39:29.076 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:39:29.080 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:39:29.080 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:39:29.414 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:39:29.414 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:39:30.739 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:39:30.848 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:32.982 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:33.267 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:33.267 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:33.267 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:39:33.267 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:33.267 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:33.267 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:39:33.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:39:34.021 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:08:55 ago on Mon 25 Mar 2024 06:30:39 PM UTC. 2024-03-25T19:39:34.171 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T19:39:34.172 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:34.172 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T19:39:34.172 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:34.172 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T19:39:34.172 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:39:34.172 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:34.173 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T19:39:34.173 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T19:39:34.173 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T19:39:34.173 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:34.173 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 1.3 M 2024-03-25T19:39:34.173 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 5.5 M 2024-03-25T19:39:34.173 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T19:39:34.675 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.5 MB/s | 1.3 MB 00:00 2024-03-25T19:39:34.676 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:34.676 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.5 MB/s | 1.3 MB 00:00 2024-03-25T19:39:34.677 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T19:39:34.680 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T19:39:34.680 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T19:39:34.723 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T19:39:34.723 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T19:39:34.963 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T19:39:35.058 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:35.307 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:39:35.397 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:35.469 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:35.743 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:35.744 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:35.744 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T19:39:35.744 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:35.744 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:35.744 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T19:39:35.918 DEBUG:teuthology.parallel:result is None 2024-03-25T19:39:37.501 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:37.770 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:37.770 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:37.770 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:39:37.770 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:37.770 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:37.770 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:39:37.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:39:38.552 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 1:08:57 ago on Mon 25 Mar 2024 06:30:41 PM UTC. 2024-03-25T19:39:38.701 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.3 M 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 5.5 M 2024-03-25T19:39:38.702 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-25T19:39:39.759 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 1.3 MB 00:01 2024-03-25T19:39:39.759 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:39.760 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.2 MB/s | 1.3 MB 00:01 2024-03-25T19:39:39.760 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-25T19:39:39.764 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-25T19:39:39.764 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-25T19:39:39.808 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-25T19:39:39.808 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-25T19:39:40.059 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-25T19:39:40.157 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:40.566 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:40.828 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:40.828 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:40.829 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-25T19:39:40.829 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:40.829 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:40.829 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-25T19:39:41.053 DEBUG:teuthology.parallel:result is None 2024-03-25T19:39:41.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:39:41.276 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:39:41.296 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:41.296 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:41.296 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:39:41.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:39:41.516 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:39:41.540 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:41.540 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:39:41.540 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:39:41.542 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:39:41.542 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:41.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:39:41.569 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:39:41.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:39:41.611 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:39:41.612 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:41.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:39:41.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:39:41.705 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:39:41.705 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:39:41.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:39:41.805 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:39:41.805 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:41.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:39:41.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:39:41.899 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:39:41.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:39:41.928 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:39:41.996 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:39:41.996 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:41.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:39:42.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:39:42.090 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:39:42.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:39:42.118 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:39:42.186 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:39:42.270 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:39:42.270 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-03-25T19:39:42.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /var/log/ceph 2024-03-25T19:39:42.299 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:39:42.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:39:42.338 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:39:42.366 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:39:42.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:39:42.406 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:39:42.443 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:39:42.444 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:39:42.444 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621326', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQCzV6lpXKIBd4N/cJBTgzahLvMCwQAhXWL0LgcI3DO1snJHDmoyF4ex80DRwIxvCTQ+hZ070f+4Ry0HY0P/PM=', 'smithi193.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCknM59JyyEIUSXB1BqSUvecxOxnQEhCaT3/GXMIpAOQkfek8WYgZoMpKoTlmZYX5bmosbyObRrIa5xHthxUpxI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:39:42.444 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:39:42.472 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:39:42.488 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:39:42.527 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:39:42.555 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:42.555 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:39:42.607 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:39:42.607 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-03-25T19:39:42.661 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:39:42.662 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:42.662 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:39:42.662 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:42.662 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:42.662 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-25 18:36:07.492709965 +0000 2024-03-25T19:39:42.662 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-25 18:29:43.401567022 +0000 2024-03-25T19:39:42.662 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-25 18:29:43.401567022 +0000 2024-03-25T19:39:42.662 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-25 18:29:43.401567022 +0000 2024-03-25T19:39:42.662 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:39:42.727 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-25T19:39:42.727 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-25T19:39:42.727 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000207902 s, 2.5 MB/s 2024-03-25T19:39:42.728 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:39:42.783 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-03-25T19:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-25 18:36:07.492709965 +0000 2024-03-25T19:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-25 18:29:43.750560538 +0000 2024-03-25T19:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-25 18:29:43.750560538 +0000 2024-03-25T19:39:42.839 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-25 18:29:43.750560538 +0000 2024-03-25T19:39:42.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:39:42.903 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-25T19:39:42.903 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-25T19:39:42.904 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000167968 s, 3.0 MB/s 2024-03-25T19:39:42.905 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:39:42.960 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-25 18:36:07.492709965 +0000 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-25 18:29:44.080554407 +0000 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-25 18:29:44.080554407 +0000 2024-03-25T19:39:43.015 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-25 18:29:44.080554407 +0000 2024-03-25T19:39:43.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:39:43.080 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-25T19:39:43.080 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-25T19:39:43.080 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000160605 s, 3.2 MB/s 2024-03-25T19:39:43.081 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:39:43.137 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-25 18:36:07.493709947 +0000 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-25 18:29:44.419548109 +0000 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-25 18:29:44.419548109 +0000 2024-03-25T19:39:43.192 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-25 18:29:44.419548109 +0000 2024-03-25T19:39:43.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:39:43.257 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-25T19:39:43.258 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-25T19:39:43.258 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000229061 s, 2.2 MB/s 2024-03-25T19:39:43.259 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:39:43.316 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:43.316 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:39:43.316 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:39:43.333 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:39:43.333 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-25 18:36:06.053549663 +0000 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-25 18:29:43.398646629 +0000 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-25 18:29:43.398646629 +0000 2024-03-25T19:39:43.390 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-25 18:29:43.398646629 +0000 2024-03-25T19:39:43.391 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:39:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-25T19:39:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-25T19:39:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000217255 s, 2.4 MB/s 2024-03-25T19:39:43.460 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:39:43.517 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-25 18:36:06.053549663 +0000 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-25 18:29:43.758641199 +0000 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-25 18:29:43.758641199 +0000 2024-03-25T19:39:43.573 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-25 18:29:43.758641199 +0000 2024-03-25T19:39:43.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:39:43.640 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-25T19:39:43.640 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-25T19:39:43.640 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000196279 s, 2.6 MB/s 2024-03-25T19:39:43.641 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:39:43.697 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-03-25T19:39:43.752 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:39:43.752 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:43.752 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:39:43.752 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:43.752 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:43.753 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-25 18:36:06.054549648 +0000 2024-03-25T19:39:43.753 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-25 18:29:44.088636222 +0000 2024-03-25T19:39:43.753 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-25 18:29:44.088636222 +0000 2024-03-25T19:39:43.753 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-25 18:29:44.088636222 +0000 2024-03-25T19:39:43.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:39:43.818 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-25T19:39:43.818 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-25T19:39:43.819 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000233476 s, 2.2 MB/s 2024-03-25T19:39:43.820 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:39:43.876 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-25 18:36:06.054549648 +0000 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-25 18:29:44.445630838 +0000 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-25 18:29:44.445630838 +0000 2024-03-25T19:39:43.931 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-25 18:29:44.445630838 +0000 2024-03-25T19:39:43.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:39:43.997 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-25T19:39:43.997 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-25T19:39:43.997 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000227624 s, 2.2 MB/s 2024-03-25T19:39:43.998 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:39:44.053 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:44.054 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:39:44.054 INFO:tasks.ceph:Generating config... 2024-03-25T19:39:44.055 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:39:44.055 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:39:44.055 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:39:44.055 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:39:44.055 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:39:44.055 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:39:44.055 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:39:44.055 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:39:44.056 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:39:44.056 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:39:44.056 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:44.057 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:39:44.058 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:39:44.059 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:39:44.059 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:39:44.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:39:44.137 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:39:44.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:39:44.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:44.211 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.71:6789]'), ('mon.c', '[v1:172.21.15.71:6790]'), ('mon.b', '[v1:172.21.15.193:6789]')] 2024-03-25T19:39:44.212 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.193:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:39:44.212 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:44.212 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:39:44.266 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.71:6789]' --addv c '[v1:172.21.15.71:6790]' --addv b '[v1:172.21.15.193:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = quincy 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 17 (quincy) 2024-03-25T19:39:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 3 2024-03-25T19:39:44.353 INFO:teuthology.orchestra.run.smithi071.stdout:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:39:44.353 INFO:teuthology.orchestra.run.smithi071.stdout:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:39:44.353 INFO:teuthology.orchestra.run.smithi071.stdout:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:39:44.353 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:39:44.354 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:39:44.368 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID acf7b4d8-7b11-4409-8ed5-296e15695260... 2024-03-25T19:39:44.369 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:39:44.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:39:44.458 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-03-25T19:39:44.458 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-03-25T19:39:44.458 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:39:44.458 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-03-25T19:39:44.458 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.458 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:39:44.459 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-03-25T19:39:44.460 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.193:6789]" 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: mon election default strategy = 3 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: ms bind msgr2 = False 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 5000 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: ms type = async 2024-03-25T19:39:44.461 INFO:teuthology.orchestra.run.smithi071.stdout: osd heartbeat grace = 80 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-03-25T19:39:44.462 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.463 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: bdev async discard = True 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: bdev enable discard = True 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore allocator = stupid 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore zero block detection = True 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 1 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 1 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:44.464 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd class default list = * 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd class load list = * 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd client watch timeout = 120 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify cached snaps = True 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify missing on start = True 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd fast shutdown = False 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd max markdown count = 1000 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock override recovery settings = True 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue = debug_random 2024-03-25T19:39:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue cut off = debug_random 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: osd shutdown pgref assert = True 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-25T19:39:44.466 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:39:44.467 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon min osdmap epochs = 50 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd crush smoke test = False 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune interval = 2 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune min = 15 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub interval = 300 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on pool no app = False 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: paxos service trim min = 10 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-03-25T19:39:44.468 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout: debug objecter = 20 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout: debug rados = 20 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2024-03-25T19:39:44.469 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout:[global] 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: chdir = "" 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: auth supported = cephx 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: filestore xattr use omap = true 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: mon clock drift allowed = 1.000 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: osd crush chooseleaf type = 0 2024-03-25T19:39:44.470 INFO:teuthology.orchestra.run.smithi193.stdout: auth debug = true 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on old message = true 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on bug = true 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: mon pg warn max object skew = 0 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:39:44.471 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default size = 2 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow primary affinity = true 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow pg remap = true 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on no sortbitwise = false 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on too few osds = false 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: mon_allow_pool_size_one = true 2024-03-25T19:39:44.472 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: osd default data pool replay window = 5 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: mon allow pool delete = true 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: mon cluster log file level = debug 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: debug asserts on shutdown = true 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: mon health detail to clog = false 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: mon host = "[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.193:6789]" 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: mon client directed command retry = 5 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: mon election default strategy = 3 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: ms bind msgr2 = False 2024-03-25T19:39:44.473 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject socket failures = 5000 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: ms type = async 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: osd heartbeat grace = 80 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: fsid = acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout:[osd] 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: osd journal size = 100 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub load threshold = 5.0 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub max interval = 600 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: osd recover clone overlap = true 2024-03-25T19:39:44.474 INFO:teuthology.orchestra.run.smithi193.stdout: osd recovery max chunk = 1048576 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug shutdown = true 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug op order = true 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify stray on activate = true 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug trim objects = true 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: osd open classes on start = true 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug pg log writeout = true 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: osd map max advance = 10 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.475 INFO:teuthology.orchestra.run.smithi193.stdout: journal zero on create = true 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: filestore apply finisher threads = 3 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: bdev debug aio = true 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug misdirected ops = true 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: bdev async discard = True 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: bdev enable discard = True 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore allocator = stupid 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore block size = 96636764160 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore fsck on mount = True 2024-03-25T19:39:44.476 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore zero block detection = True 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: debug bluefs = 1 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: debug bluestore = 1 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: debug monc = 20 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: debug osd = 20 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: debug rocksdb = 10 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd full ratio = 0.9 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:39:44.477 INFO:teuthology.orchestra.run.smithi193.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd class default list = * 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd class load list = * 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd client watch timeout = 120 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify cached snaps = True 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify missing on start = True 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd fast shutdown = False 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd max markdown count = 1000 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd mclock override recovery settings = True 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd objectstore = bluestore 2024-03-25T19:39:44.478 INFO:teuthology.orchestra.run.smithi193.stdout: osd op queue = debug_random 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: osd op queue cut off = debug_random 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: osd shutdown pgref assert = True 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr] 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: debug mgr = 20 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:44.479 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: mgr/telemetry/nag = false 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout:[mon] 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: debug paxos = 20 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: mon data avail warn = 5 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: mon mgr mkfs grace = 240 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:39:44.480 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd reporter subtree level = osd 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd prime pg temp = true 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: # 1m isn't quite enough 2024-03-25T19:39:44.481 INFO:teuthology.orchestra.run.smithi193.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: mon min osdmap epochs = 50 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd crush smoke test = False 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: mon osdmap full prune interval = 2 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: mon osdmap full prune min = 15 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: mon scrub interval = 300 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on pool no app = False 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: paxos service trim min = 10 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout:[client] 2024-03-25T19:39:44.482 INFO:teuthology.orchestra.run.smithi193.stdout: rgw cache enabled = true 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable ops log = true 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable usage log = true 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout: debug monc = 20 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout: debug objecter = 20 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout: debug rados = 20 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.a] 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.c] 2024-03-25T19:39:44.483 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.b] 2024-03-25T19:39:44.484 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:39:44.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:39:44.541 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:39:44.541 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:44.541 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:39:44.556 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:44.556 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:39:44.610 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:39:44.610 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:44.610 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:39:44.611 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:44.690 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:39:44.690 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:44.745 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:39:44.745 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:39:44.745 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:39:44.745 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:39:44.784 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:39:44.785 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:44.839 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:39:44.839 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:39:44.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:39:44.894 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:39:44.896 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:39:44.971 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:39:44.973 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:39:44.973 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:39:44.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:39:45.016 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:39:45.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:39:45.073 INFO:teuthology.orchestra.run.smithi193.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:39:45.088 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:39:45.089 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:39:45.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:45.115 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:45.116 INFO:tasks.ceph:role: osd.0 2024-03-25T19:39:45.116 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:39:45.116 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:39:45.190 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:45.191 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:45.698 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-25T19:39:45.700 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-25T19:39:45.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:45.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:45.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:45.949 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:45.949 INFO:tasks.ceph:role: osd.1 2024-03-25T19:39:45.949 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:39:45.950 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:39:46.023 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:46.023 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:46.024 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:46.024 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:46.024 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:46.024 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:46.024 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:46.024 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:46.024 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:46.024 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:46.472 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-25T19:39:46.473 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-25T19:39:46.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:46.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:46.599 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:46.667 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:46.667 INFO:tasks.ceph:role: osd.2 2024-03-25T19:39:46.667 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:39:46.667 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:39:46.742 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:46.742 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:46.742 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:46.742 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:46.742 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:46.742 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:46.742 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:46.743 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:46.743 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:46.743 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:47.230 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-25T19:39:47.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-25T19:39:47.232 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:47.369 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:47.400 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:47.467 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:39:47.467 INFO:tasks.ceph:role: osd.3 2024-03-25T19:39:47.467 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:39:47.467 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:39:47.541 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:39:47.541 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:39:47.541 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:39:47.541 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:39:47.541 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:39:47.542 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:39:47.542 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:39:47.542 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:39:47.542 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:39:47.542 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:39:48.066 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-25T19:39:48.068 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-25T19:39:48.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:48.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:39:48.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:48.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:48.360+0000 7f0300c0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:39:48.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:48.360+0000 7f0300c0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:39:48.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:48.360+0000 7f0300c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:48.360 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:48.360+0000 7f0300c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:48.361 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:48.360+0000 7f0300c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:39:48.361 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:48.361+0000 7f0300c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:39:51.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:39:51.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:51.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:51.330+0000 7f4701f1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:39:51.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:51.330+0000 7f4701f1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:39:51.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:51.330+0000 7f4701f1f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:51.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:51.330+0000 7f4701f1f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:51.330 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:51.330+0000 7f4701f1f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:39:51.331 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:51.331+0000 7f4701f1f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:39:54.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:39:54.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:54.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:54.292+0000 7ffa96547640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:39:54.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:54.292+0000 7ffa96547640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:39:54.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:54.292+0000 7ffa96547640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:54.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:54.292+0000 7ffa96547640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:54.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:54.292+0000 7ffa96547640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:39:54.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:54.293+0000 7ffa96547640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:39:57.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:39:57.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:39:57.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:57.269+0000 7f268b8d5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:39:57.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:57.269+0000 7f268b8d5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:39:57.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:57.269+0000 7f268b8d5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:39:57.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:57.269+0000 7f268b8d5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:39:57.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:57.269+0000 7f268b8d5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:39:57.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:39:57.269+0000 7f268b8d5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:40:00.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:00.141 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:00.173 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:00.173 INFO:tasks.ceph:role: osd.4 2024-03-25T19:40:00.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:40:00.174 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:40:00.251 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:00.251 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:00.251 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:00.251 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:00.251 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:00.251 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:00.252 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:00.252 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:00.252 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:00.252 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:00.763 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-03-25T19:40:00.765 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-03-25T19:40:00.766 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:00.909 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:00.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:01.016 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:01.017 INFO:tasks.ceph:role: osd.5 2024-03-25T19:40:01.017 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:40:01.017 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:01.092 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:01.093 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:01.539 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-03-25T19:40:01.541 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-03-25T19:40:01.541 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:01.640 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:01.673 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:01.741 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:01.742 INFO:tasks.ceph:role: osd.6 2024-03-25T19:40:01.742 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:40:01.742 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:01.819 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:01.819 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:01.819 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:01.819 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:02.314 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-03-25T19:40:02.315 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-03-25T19:40:02.315 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:02.450 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:02.485 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:02.554 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:02.554 INFO:tasks.ceph:role: osd.7 2024-03-25T19:40:02.554 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:40:02.554 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:40:02.629 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:02.630 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:02.630 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:02.630 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:02.630 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:03.104 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-03-25T19:40:03.106 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-03-25T19:40:03.106 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:03.235 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:03.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:03.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:03.369+0000 7f9099212640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:40:03.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:03.369+0000 7f9099212640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:40:03.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:03.369+0000 7f9099212640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:03.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:03.369+0000 7f9099212640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:03.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:03.369+0000 7f9099212640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:03.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:03.370+0000 7f9099212640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:40:06.214 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:06.244 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:06.334 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:06.334+0000 7ffbe7b91640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:40:06.334 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:06.334+0000 7ffbe7b91640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:40:06.334 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:06.334+0000 7ffbe7b91640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:06.334 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:06.334+0000 7ffbe7b91640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:06.334 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:06.334+0000 7ffbe7b91640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:06.335 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:06.335+0000 7ffbe7b91640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:40:09.171 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:09.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:09.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:09.294+0000 7f026f581640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:40:09.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:09.295+0000 7f026f581640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:40:09.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:09.295+0000 7f026f581640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:09.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:09.295+0000 7f026f581640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:09.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:09.295+0000 7f026f581640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:09.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:09.295+0000 7f026f581640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:40:12.144 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:12.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:12.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:12.267+0000 7f0b87981640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:40:12.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:12.267+0000 7f0b87981640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:40:12.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:12.267+0000 7f0b87981640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:12.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:12.267+0000 7f0b87981640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:12.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:12.267+0000 7f0b87981640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:12.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:12.268+0000 7f0b87981640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:40:15.115 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:15.146 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:40:15.146 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:15.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:40:15.178 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:15.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:40:15.248 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:15.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:40:15.317 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:15.317 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:40:15.385 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:15.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:40:15.454 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:15.454 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:40:15.485 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:15.485 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:40:15.556 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:15.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:40:15.625 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:15.625 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:40:15.694 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:15.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:40:15.762 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:15.762 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:40:15.778 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:15.779 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:40:15.818 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:40:15.818 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:40:15.820 DEBUG:teuthology.orchestra.run.smithi193:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:40:15.861 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.y] 2024-03-25T19:40:15.861 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAA0wFmEIZjNRAAU5/7XVMOXVHIE0RylpqaNA== 2024-03-25T19:40:15.861 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-03-25T19:40:15.861 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAE0wFmTN2LFRAAlijAsIjnsr48l2GFERVedA== 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAH0wFmQJPCExAAnW1Bo1mwrxSPpqhyZ4HW8Q== 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAK0wFmj913ERAAIi7T4tgA+9W6zsin1EQkJg== 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAN0wFmq9MWEBAAafz5ru9Mpv/KuCqN3WiDeA== 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAA0wFmmin3ORAAQdw5UoU7hp0LviYH7g6MRw== 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAT0wFmYe8NFhAA+s4W8tw+O3q7EN6AUKzgsA== 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAW0wFm/xf9ExAAfqbEgr+PFrUC/2ePVBLVDA== 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.6] 2024-03-25T19:40:15.862 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAZ0wFm9GWhERAA9InU5yzdOZp9HSIGqzEtww== 2024-03-25T19:40:15.863 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.7] 2024-03-25T19:40:15.863 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAc0wFmchH+DxAAYvdiEH3CJPc4UWx3svGbKQ== 2024-03-25T19:40:15.863 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-03-25T19:40:15.863 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAB0wFm8hsQARAArl01x+IqPXPrTm/CR6f3nA== 2024-03-25T19:40:15.863 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2024-03-25T19:40:15.863 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAB0wFmc0R2BBAAfLKjtCcQtordh6Jla+xIqw== 2024-03-25T19:40:15.884 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.y] 2024-03-25T19:40:15.884 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAA0wFmEIZjNRAAU5/7XVMOXVHIE0RylpqaNA== 2024-03-25T19:40:15.884 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.0] 2024-03-25T19:40:15.884 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAE0wFmTN2LFRAAlijAsIjnsr48l2GFERVedA== 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.1] 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAH0wFmQJPCExAAnW1Bo1mwrxSPpqhyZ4HW8Q== 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.2] 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAK0wFmj913ERAAIi7T4tgA+9W6zsin1EQkJg== 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.3] 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAN0wFmq9MWEBAAafz5ru9Mpv/KuCqN3WiDeA== 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.x] 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAA0wFmmin3ORAAQdw5UoU7hp0LviYH7g6MRw== 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.4] 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAT0wFmYe8NFhAA+s4W8tw+O3q7EN6AUKzgsA== 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.5] 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAW0wFm/xf9ExAAfqbEgr+PFrUC/2ePVBLVDA== 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.6] 2024-03-25T19:40:15.885 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAZ0wFm9GWhERAA9InU5yzdOZp9HSIGqzEtww== 2024-03-25T19:40:15.886 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.7] 2024-03-25T19:40:15.886 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAc0wFmchH+DxAAYvdiEH3CJPc4UWx3svGbKQ== 2024-03-25T19:40:15.886 INFO:teuthology.orchestra.run.smithi193.stdout:[client.0] 2024-03-25T19:40:15.886 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAB0wFm8hsQARAArl01x+IqPXPrTm/CR6f3nA== 2024-03-25T19:40:15.886 INFO:teuthology.orchestra.run.smithi193.stdout:[client.1] 2024-03-25T19:40:15.886 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAB0wFmc0R2BBAAfLKjtCcQtordh6Jla+xIqw== 2024-03-25T19:40:15.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:15.907 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:15.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:15.982 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.089 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.142 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:16.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:16.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.300 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.355 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.406 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:16.464 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:16.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:16.520 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:16.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:16.576 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:40:16.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:40:16.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:16.698 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:40:16.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:40:16.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:16.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:40:16.913 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:40:16.941 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:17.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:40:17.064 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:17.066 DEBUG:teuthology.orchestra.run.smithi193:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:17.118 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:40:17.119 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:17.119 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:40:17.119 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:40:17.122 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:40:17.122 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:17.122 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:40:17.122 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:40:17.124 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:40:17.124 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:17.124 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:40:17.124 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:40:17.161 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:40:17.162 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:40:17.162 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:40:17.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:40:17.164 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:40:17.164 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:40:17.165 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:40:17.166 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:40:17.167 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:40:17.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.277+0000 7f70cddaf640 10 monclient: get_monmap_and_config 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.277+0000 7f70cddaf640 10 monclient: build_initial_monmap 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.277+0000 7f70cddaf640 10 monclient: monmap: 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:40:17.278646+0000 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:40:17.278646+0000 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:17.277 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:17.278 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:17.278 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:17.278 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:17.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.278+0000 7f70cddaf640 1 Processor -- start 2024-03-25T19:40:17.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.278+0000 7f70cddaf640 1 -- start start 2024-03-25T19:40:17.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.278+0000 7f70cddaf640 10 monclient: init 2024-03-25T19:40:17.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:17.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:17.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 10 monclient(hunting): picked mon.noname-a con 0x7f70c80a9920 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 10 monclient(hunting): picked mon.noname-c con 0x7f70c80b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 10 monclient(hunting): picked mon.noname-b con 0x7f70c8146450 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c806b100 con 0x7f70c80a9920 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80b2940 con 0x7f70c80b45a0 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c813fec0 con 0x7f70c8146450 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70cddaf640 10 monclient(hunting): authenticate will time out at 4823.591024s 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70c80a9920 legacy=0x7f70c813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70c7fff640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70c814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:17.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.279+0000 7f70c6ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70c8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:17.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.480+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70c80a9920 legacy=0x7f70c813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:17.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.480+0000 7f70c7fff640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70c814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:17.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.480+0000 7f70c6ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70c8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:17.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.881+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70c80a9920 legacy=0x7f70c813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:17.881 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.881+0000 7f70c7fff640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70c814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:17.881 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:17.881+0000 7f70c6ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70c8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:18.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:18.682+0000 7f70c7fff640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70c814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:18.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:18.682+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70c80a9920 legacy=0x7f70c813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:18.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:18.682+0000 7f70c6ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70c8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:20.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 10 monclient(hunting): tick 2024-03-25T19:40:20.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:20.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:20.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:20.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70c814dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70c8145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70c80a9920 legacy=0x7f70c813f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f70a8003680 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f70c80b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f70c8146450 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c806b100 con 0x7f70a8003680 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80b2940 con 0x7f70c80b45a0 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c813fec0 con 0x7f70c8146450 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c57fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c7fff640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70a800cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:20.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.279+0000 7f70c6ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:20.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.480+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:20.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.480+0000 7f70c7fff640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70a800cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:20.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.480+0000 7f70c6ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:20.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.881+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:20.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.881+0000 7f70c7fff640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70a800cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:20.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:20.881+0000 7f70c6ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:21.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:21.682+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:21.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:21.682+0000 7f70c7fff640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70a800cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:21.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:21.682+0000 7f70c6ffd640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:23.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 10 monclient(hunting): tick 2024-03-25T19:40:23.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:23.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:23.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:23.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70c8146450 legacy=0x7f70a800cba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:23.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:23.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f70a8010340 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f70a8003680 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f70c80b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8011350 con 0x7f70a8003680 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c806b100 con 0x7f70c80b45a0 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80b2940 con 0x7f70a8010340 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c57fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70a8010340 legacy=0x7f70a8010720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c6ffd640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:23.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.279+0000 7f70c7fff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.480+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70a8010340 legacy=0x7f70a8010720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.480+0000 7f70c6ffd640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:23.479 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.480+0000 7f70c7fff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:23.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.881+0000 7f70c6ffd640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:23.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.881+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70a8010340 legacy=0x7f70a8010720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:23.880 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:23.881+0000 7f70c7fff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:24.140 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:00:06.859 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:24.304 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:00:07.060 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:24.306 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:00:07.062 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:24.447 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:00:07.166 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:24.624 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:00:07.381 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:24.626 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:00:07.383 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:24.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:24.682+0000 7f70c6ffd640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-03-25T19:40:24.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:24.682+0000 7f70c77fe640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70a8010340 legacy=0x7f70a8010720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6790/0 2024-03-25T19:40:24.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:24.682+0000 7f70c7fff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T19:40:26.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 10 monclient(hunting): tick 2024-03-25T19:40:26.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.71:6790/0 conn(0x7f70a8010340 legacy=0x7f70a8010720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f70a8003680 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f70a800d2d0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f70a8010340 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f70c80b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c813fec0 con 0x7f70a800d2d0 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8011350 con 0x7f70c80b45a0 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c806b100 con 0x7f70a8010340 2024-03-25T19:40:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.279+0000 7f70c57fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:26.322 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.323+0000 7f70c7fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 0x7f70a80093a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:57046/0 (socket says 172.21.15.71:57046) 2024-03-25T19:40:26.322 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:26.323+0000 7f70c7fff640 1 -- 172.21.15.71:0/1703597681 learned_addr learned my addr 172.21.15.71:0/1703597681 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:40:29.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.279+0000 7f70c57fa640 10 monclient(hunting): tick 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.279+0000 7f70c57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.279+0000 7f70c57fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.279+0000 7f70c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.279+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6790/0 conn(0x7f70a8010340 legacy=0x7f70a8005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.279+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.279+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6789/0 conn(0x7f70a800d2d0 legacy=0x7f70a800cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.280+0000 7f70c57fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f70a8010720 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f70a800fdd0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f70c80b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80b2940 con 0x7f70a800fdd0 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8010bb0 con 0x7f70c80b45a0 2024-03-25T19:40:29.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8005ae0 con 0x7f70a8010720 2024-03-25T19:40:29.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:29.280+0000 7f70c57fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:32.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 10 monclient(hunting): tick 2024-03-25T19:40:32.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:32.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:32.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:32.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6790/0 conn(0x7f70a8010720 legacy=0x7f70a800f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.193:6789/0 conn(0x7f70c80b45a0 legacy=0x7f70a8009ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6789/0 conn(0x7f70a800fdd0 legacy=0x7f70a80155f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f70a800a1e0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f70a800a5c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f70a8010720 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8005cc0 con 0x7f70a800a5c0 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a800cbb0 con 0x7f70a800a1e0 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a800cd40 con 0x7f70a8010720 2024-03-25T19:40:32.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:32.280+0000 7f70c57fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:35.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 10 monclient(hunting): tick 2024-03-25T19:40:35.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:35.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:35.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6790/0 conn(0x7f70a8010720 legacy=0x7f70a800f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.193:6789/0 conn(0x7f70a800a1e0 legacy=0x7f70a8003ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6789/0 conn(0x7f70a800a5c0 legacy=0x7f70a80093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f70a8009ad0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f70a8003a90 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f70a8010720 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a800ced0 con 0x7f70a8003a90 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a800d110 con 0x7f70a8009ad0 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8009f90 con 0x7f70a8010720 2024-03-25T19:40:35.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:35.280+0000 7f70c57fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:38.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 10 monclient(hunting): tick 2024-03-25T19:40:38.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:38.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:38.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:38.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6790/0 conn(0x7f70a8010720 legacy=0x7f70a800f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:38.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.193:6789/0 conn(0x7f70a8009ad0 legacy=0x7f70a80051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6789/0 conn(0x7f70a8003a90 legacy=0x7f70a8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f70a8009540 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f70a8010720 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f70a8003a90 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80093b0 con 0x7f70a8003a90 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80053a0 con 0x7f70a8009540 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8005530 con 0x7f70a8010720 2024-03-25T19:40:38.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:38.281+0000 7f70c57fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:39.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.212+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397694560 0 0) 0x7f70a80093b0 con 0x7f70a8003a90 2024-03-25T19:40:39.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.212+0000 7f70c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.212+0000 7f70c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.212+0000 7f70c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.212+0000 7f70c67fc640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:40:39.212 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.212+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b4003520 con 0x7f70a8003a90 2024-03-25T19:40:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.246+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526468149 0 0) 0x7f70a80053a0 con 0x7f70a8009540 2024-03-25T19:40:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.246+0000 7f70c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.246+0000 7f70c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.246+0000 7f70c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.246+0000 7f70c67fc640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:40:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.246+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a80093b0 con 0x7f70a8009540 2024-03-25T19:40:39.825 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1906052891 0 0) 0x7f70a80093b0 con 0x7f70a8009540 2024-03-25T19:40:39.825 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6790/0 conn(0x7f70a8010720 legacy=0x7f70a8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.825 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.71:6789/0 conn(0x7f70a8003a90 legacy=0x7f70a8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.825 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 1 monclient: found mon.noname-c 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a8005710 con 0x7f70a8009540 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f70b8003ee0 con 0x7f70a8009540 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70c67fc640 10 monclient: dump: 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:39.826 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70cddaf640 5 monclient: authenticate success, global_id 4105 2024-03-25T19:40:39.827 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.825+0000 7f70cddaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:39.852 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.852+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b8004040 con 0x7f70a8009540 2024-03-25T19:40:39.852 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.852+0000 7f70c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:39.852 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.852+0000 7f70c67fc640 1 -- 172.21.15.71:0/1703597681 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f70b8004490 con 0x7f70a8009540 2024-03-25T19:40:39.852 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.852+0000 7f70cddaf640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:39.852 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.852+0000 7f70cddaf640 4 set_mon_vals no callback set 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.852+0000 7f70c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.852+0000 7f70c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.852+0000 7f70c67fc640 10 monclient: dump: 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:39.853 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.853+0000 7f70cddaf640 10 monclient: shutdown 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.853+0000 7f70cddaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.853+0000 7f70cddaf640 1 -- 172.21.15.71:0/1703597681 >> v1:172.21.15.193:6789/0 conn(0x7f70a8009540 legacy=0x7f70a8004580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.853+0000 7f70cddaf640 1 -- 172.21.15.71:0/1703597681 shutdown_connections 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.853+0000 7f70cddaf640 1 -- 172.21.15.71:0/1703597681 >> 172.21.15.71:0/1703597681 conn(0x7f70c80b7f40 msgr2=0x7f70c80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.853+0000 7f70cddaf640 1 -- 172.21.15.71:0/1703597681 shutdown_connections 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.853+0000 7f70cddaf640 1 -- 172.21.15.71:0/1703597681 wait complete. 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.854+0000 7f70cddaf640 10 monclient: build_initial_monmap 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.854+0000 7f70cddaf640 10 monclient: monmap: 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.854+0000 7f70cddaf640 1 librados: starting msgr at 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.854+0000 7f70cddaf640 1 librados: starting objecter 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.854+0000 7f70cddaf640 1 Processor -- start 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 1 -- start start 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 1 librados: setting wanted keys 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 1 librados: calling monclient init 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 10 monclient: init 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 10 monclient(hunting): picked mon.noname-a con 0x7f70a8003a90 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 10 monclient(hunting): picked mon.noname-b con 0x7f70a8009540 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 10 monclient(hunting): picked mon.noname-c con 0x7f70a8010720 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80ad150 con 0x7f70a8003a90 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c81daa20 con 0x7f70a8009540 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80ad3c0 con 0x7f70a8010720 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:39.858 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.855+0000 7f70cddaf640 10 monclient(hunting): authenticate will time out at 4846.167542s 2024-03-25T19:40:39.861 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.862+0000 7f70c7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f70a8010720 0x7f70c81e2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52854/0 (socket says 172.21.15.71:52854) 2024-03-25T19:40:39.861 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.862+0000 7f70c7fff640 1 -- 172.21.15.71:0/3371219083 learned_addr learned my addr 172.21.15.71:0/3371219083 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:40:39.863 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.864+0000 7f70bffff640 10 client.?.objecter ms_handle_connect 0x7f70a8010720 2024-03-25T19:40:39.864 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.864+0000 7f70bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:39.865 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.865+0000 7f70bffff640 10 client.?.objecter ms_handle_connect 0x7f70a8009540 2024-03-25T19:40:39.865 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.865+0000 7f70bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:39.866 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.867+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026765816 0 0) 0x7f70c80ad3c0 con 0x7f70a8010720 2024-03-25T19:40:39.866 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.867+0000 7f70bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.866 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.867+0000 7f70bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.866 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.867+0000 7f70bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.867+0000 7f70bffff640 10 monclient(hunting): my global_id is 14108 2024-03-25T19:40:39.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.867+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a800d110 con 0x7f70a8010720 2024-03-25T19:40:39.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.868+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214152398 0 0) 0x7f70c81daa20 con 0x7f70a8009540 2024-03-25T19:40:39.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.868+0000 7f70bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.868+0000 7f70bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.868+0000 7f70bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.867 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.868+0000 7f70bffff640 10 monclient(hunting): my global_id is 4108 2024-03-25T19:40:39.868 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.868+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c80ad3c0 con 0x7f70a8009540 2024-03-25T19:40:39.881 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.881+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1198997837 0 0) 0x7f70c80ad3c0 con 0x7f70a8009540 2024-03-25T19:40:39.881 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.881+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70c81daa20 con 0x7f70a8009540 2024-03-25T19:40:39.881 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.881+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f70b0004520 con 0x7f70a8009540 2024-03-25T19:40:39.881 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.882+0000 7f70bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.881 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.882+0000 7f70bffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:39.881 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.882+0000 7f70bffff640 10 monclient(hunting): dump: 2024-03-25T19:40:39.881 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:39.882 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.882+0000 7f70bffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.892+0000 7f70bffff640 10 client.?.objecter ms_handle_connect 0x7f70a8003a90 2024-03-25T19:40:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.892+0000 7f70bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:39.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 492761400 0 0) 0x7f70c81daa20 con 0x7f70a8009540 2024-03-25T19:40:39.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 >> v1:172.21.15.71:6790/0 conn(0x7f70a8010720 legacy=0x7f70c81e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 >> v1:172.21.15.71:6789/0 conn(0x7f70a8003a90 legacy=0x7f70c81daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:39.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 1 monclient: found mon.b 2024-03-25T19:40:39.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c80ac510 con 0x7f70a8009540 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 5 monclient: authenticate success, global_id 4108 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 10 monclient: _renew_subs 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c8140730 con 0x7f70a8009540 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 10 monclient: _renew_subs 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c81db7f0 con 0x7f70a8009540 2024-03-25T19:40:39.897 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.896+0000 7f70cddaf640 1 librados: init done 2024-03-25T19:40:39.898 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.897+0000 7f70cddaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:39.898 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.897+0000 7f70cddaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:39.898 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.897+0000 7f70cddaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:39.898 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:39.897+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c8140920 con 0x7f70a8009540 2024-03-25T19:40:40.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.090+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70b0004930 con 0x7f70a8009540 2024-03-25T19:40:40.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.090+0000 7f70bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:40.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.090+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f70b0004d40 con 0x7f70a8009540 2024-03-25T19:40:40.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.090+0000 7f70bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:40.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.090+0000 7f70bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:40.090 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.090+0000 7f70bffff640 10 monclient: dump: 2024-03-25T19:40:40.090 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.090+0000 7f70c5ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:40.091 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.090+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f70b00052f0 con 0x7f70a8009540 2024-03-25T19:40:40.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.093+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3247443069 0 0) 0x7f70b0006570 con 0x7f70a8009540 2024-03-25T19:40:40.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.093+0000 7f70bffff640 10 client.4108.objecter ms_dispatch 0x7f70c8000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:40:40.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.093+0000 7f70bffff640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:40:40.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.093+0000 7f70bffff640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:40:40.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.093+0000 7f70bffff640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-25T19:40:40.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.093+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f70b0006b70 con 0x7f70a8009540 2024-03-25T19:40:40.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.093+0000 7f70bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:40.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.093+0000 7f70bffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.180+0000 7f70cddaf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.180+0000 7f70cddaf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.180+0000 7f70cddaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.180+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f70c8140920 con 0x7f70a8009540 2024-03-25T19:40:40.443 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:40.444+0000 7efe2da0e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:40.446 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:40.447+0000 7f04fdc42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:40.500 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:40.500+0000 7f04fdc42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:40.502 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:40.503+0000 7efe2da0e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:40.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.528+0000 7f70bffff640 1 -- 172.21.15.71:0/3371219083 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f70b00058b0 con 0x7f70a8009540 2024-03-25T19:40:40.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.528+0000 7f70bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:40.528 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.528+0000 7f70bffff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:40:40.528 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2024-03-25T19:40:40.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 10 librados: watch_flush enter 2024-03-25T19:40:40.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 10 librados: watch_flush exit 2024-03-25T19:40:40.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 10 client.4108.objecter successfully canceled tick 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 10 monclient: shutdown 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 >> v1:172.21.15.193:6789/0 conn(0x7f70a8009540 legacy=0x7f70c8137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 shutdown_connections 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 >> 172.21.15.71:0/3371219083 conn(0x7f70c80b7f40 msgr2=0x7f70c8145310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 shutdown_connections 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 1 -- 172.21.15.71:0/3371219083 wait complete. 2024-03-25T19:40:40.530 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:40.529+0000 7f70cddaf640 1 librados: shutdown 2024-03-25T19:40:40.540 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:40:40.540 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:40:40.540 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:40.540 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:40:40.567 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:40.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:40:40.634 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:40.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:40:40.701 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-25T19:40:40.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:40:40.771 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:40.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:40:40.799 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:40.799 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:40:40.868 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:40.868 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:40:40.935 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T19:40:40.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:40:41.003 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new d573c566-53b7-4309-b868-cd9103387d76 0 2024-03-25T19:40:41.072 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.073+0000 7f04fdc42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:41.088 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.089+0000 7efe2da0e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:41.129 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.130+0000 7f04fdc42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:41.142 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.142+0000 7efe2da0e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:41.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.146+0000 7fbc0d332640 10 monclient: get_monmap_and_config 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.146+0000 7fbc0d332640 10 monclient: build_initial_monmap 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.146+0000 7fbc0d332640 10 monclient: monmap: 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:40:41.147397+0000 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:40:41.147397+0000 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:41.146 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.147 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 1 Processor -- start 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 1 -- start start 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 10 monclient: init 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 10 monclient(hunting): picked mon.noname-b con 0x7fbc080a9920 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 10 monclient(hunting): picked mon.noname-c con 0x7fbc080b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 10 monclient(hunting): picked mon.noname-a con 0x7fbc08146450 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0806b100 con 0x7fbc08146450 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc080b2940 con 0x7fbc080b45a0 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0813fec0 con 0x7fbc080a9920 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.147+0000 7fbc0d332640 10 monclient(hunting): authenticate will time out at 4849.300362s 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.148+0000 7fbc067fc640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fbc080b45a0 0x7fbc08145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52628/0 (socket says 172.21.15.193:52628) 2024-03-25T19:40:41.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.148+0000 7fbc067fc640 1 -- 172.21.15.193:0/350016566 learned_addr learned my addr 172.21.15.193:0/350016566 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186902618 0 0) 0x7fbc080b2940 con 0x7fbc080b45a0 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf0003520 con 0x7fbc080b45a0 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043434054 0 0) 0x7fbc0813fec0 con 0x7fbc080a9920 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 10 monclient(hunting): my global_id is 14111 2024-03-25T19:40:41.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.150+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc080b2940 con 0x7fbc080a9920 2024-03-25T19:40:41.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.152+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288737056 0 0) 0x7fbc0806b100 con 0x7fbc08146450 2024-03-25T19:40:41.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.152+0000 7fbc05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.152+0000 7fbc05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.152+0000 7fbc05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.152+0000 7fbc05ffb640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:40:41.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.152+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0813fec0 con 0x7fbc08146450 2024-03-25T19:40:41.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453930834 0 0) 0x7fbbf0003520 con 0x7fbc080b45a0 2024-03-25T19:40:41.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 >> v1:172.21.15.71:6790/0 conn(0x7fbc080a9920 legacy=0x7fbc0813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 >> v1:172.21.15.71:6789/0 conn(0x7fbc08146450 legacy=0x7fbc0814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 1 monclient: found mon.noname-c 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc08140050 con 0x7fbc080b45a0 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbbfc001690 con 0x7fbc080b45a0 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:41.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc05ffb640 10 monclient: dump: 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc0d332640 5 monclient: authenticate success, global_id 4117 2024-03-25T19:40:41.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.154+0000 7fbc0d332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.155+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbfc0017f0 con 0x7fbc080b45a0 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.155+0000 7fbc05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.155+0000 7fbc05ffb640 1 -- 172.21.15.193:0/350016566 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbbfc001a60 con 0x7fbc080b45a0 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.155+0000 7fbc05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.155+0000 7fbc05ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.155+0000 7fbc05ffb640 10 monclient: dump: 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.157 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.155+0000 7fbc0d332640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.155+0000 7fbc0d332640 4 set_mon_vals no callback set 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.156+0000 7fbc0d332640 10 monclient: shutdown 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.156+0000 7fbc0d332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.156+0000 7fbc0d332640 1 -- 172.21.15.193:0/350016566 >> v1:172.21.15.193:6789/0 conn(0x7fbc080b45a0 legacy=0x7fbc08145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.156+0000 7fbc0d332640 1 -- 172.21.15.193:0/350016566 shutdown_connections 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.156+0000 7fbc0d332640 1 -- 172.21.15.193:0/350016566 >> 172.21.15.193:0/350016566 conn(0x7fbc080b7f40 msgr2=0x7fbc080b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.156+0000 7fbc0d332640 1 -- 172.21.15.193:0/350016566 shutdown_connections 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.156+0000 7fbc0d332640 1 -- 172.21.15.193:0/350016566 wait complete. 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.157+0000 7fbc0d332640 10 monclient: build_initial_monmap 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.157+0000 7fbc0d332640 10 monclient: monmap: 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:41.159 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.157+0000 7fbc0d332640 1 librados: starting msgr at 2024-03-25T19:40:41.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.157+0000 7fbc0d332640 1 librados: starting objecter 2024-03-25T19:40:41.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 1 Processor -- start 2024-03-25T19:40:41.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 1 -- start start 2024-03-25T19:40:41.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 1 librados: setting wanted keys 2024-03-25T19:40:41.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 1 librados: calling monclient init 2024-03-25T19:40:41.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 10 monclient: init 2024-03-25T19:40:41.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:41.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 10 monclient(hunting): picked mon.noname-b con 0x7fbc080a9920 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 10 monclient(hunting): picked mon.noname-c con 0x7fbc080b45a0 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 10 monclient(hunting): picked mon.noname-a con 0x7fbc08146450 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc080ac400 con 0x7fbc08146450 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc080ac650 con 0x7fbc080a9920 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc081448f0 con 0x7fbc080b45a0 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.158+0000 7fbc0d332640 10 monclient(hunting): authenticate will time out at 4849.311425s 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.159+0000 7fbc067fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fbc080b45a0 0x7fbc080b11e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.193:51492/0 (socket says 172.21.15.193:51492) 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.159+0000 7fbc067fc640 1 -- 172.21.15.193:0/651030988 learned_addr learned my addr 172.21.15.193:0/651030988 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.159+0000 7fbbef7fe640 10 client.?.objecter ms_handle_connect 0x7fbc080b45a0 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.159+0000 7fbbef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.159+0000 7fbbef7fe640 10 client.?.objecter ms_handle_connect 0x7fbc08146450 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.159+0000 7fbbef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.160+0000 7fbbef7fe640 10 client.?.objecter ms_handle_connect 0x7fbc080a9920 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.160+0000 7fbbef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177257302 0 0) 0x7fbc081448f0 con 0x7fbc080b45a0 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 10 monclient(hunting): my global_id is 14114 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf4004eb0 con 0x7fbc080b45a0 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348506607 0 0) 0x7fbc080ac400 con 0x7fbc08146450 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.161+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc081448f0 con 0x7fbc08146450 2024-03-25T19:40:41.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.162+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417350663 0 0) 0x7fbc080ac650 con 0x7fbc080a9920 2024-03-25T19:40:41.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.162+0000 7fbbef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.162+0000 7fbbef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.162+0000 7fbbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.162+0000 7fbbef7fe640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:40:41.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.162+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc080ac400 con 0x7fbc080a9920 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1324624552 0 0) 0x7fbbf4004eb0 con 0x7fbc080b45a0 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc080ac650 con 0x7fbc080b45a0 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbbfc001b00 con 0x7fbc080b45a0 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.166 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3259801509 0 0) 0x7fbc080ac400 con 0x7fbc080a9920 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbf4004eb0 con 0x7fbc080a9920 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbbf80018f0 con 0x7fbc080a9920 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:41.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.166+0000 7fbbef7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.168 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.167+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 291185972 0 0) 0x7fbc081448f0 con 0x7fbc08146450 2024-03-25T19:40:41.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.167+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc080ac400 con 0x7fbc08146450 2024-03-25T19:40:41.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.167+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbc00001b50 con 0x7fbc08146450 2024-03-25T19:40:41.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.167+0000 7fbbef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.167+0000 7fbbef7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:41.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.167+0000 7fbbef7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.169+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4151873257 0 0) 0x7fbbf4004eb0 con 0x7fbc080a9920 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 >> v1:172.21.15.71:6790/0 conn(0x7fbc080b45a0 legacy=0x7fbc080b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 >> v1:172.21.15.71:6789/0 conn(0x7fbc08146450 legacy=0x7fbc081e2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 1 monclient: found mon.b 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc08145520 con 0x7fbc080a9920 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbbef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 5 monclient: authenticate success, global_id 4120 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 10 monclient: _renew_subs 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc08144460 con 0x7fbc080a9920 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:41.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 10 monclient: _renew_subs 2024-03-25T19:40:41.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:41.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc081db9d0 con 0x7fbc080a9920 2024-03-25T19:40:41.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 1 librados: init done 2024-03-25T19:40:41.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:41.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:41.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:41.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.170+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc08144a80 con 0x7fbc080a9920 2024-03-25T19:40:41.179 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.179+0000 7f04fdc42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:41.193 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.193+0000 7efe2da0e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:41.233 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.234+0000 7f04fdc42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:41.246 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.247+0000 7efe2da0e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf80035f0 con 0x7fbc080a9920 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbbf8003900 con 0x7fbc080a9920 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 10 monclient: dump: 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.267 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbbf8003df0 con 0x7fbc080a9920 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2687781867 0 0) 0x7fbbf8005250 con 0x7fbc080a9920 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 10 client.4120.objecter ms_dispatch 0x7fbc08000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbc057fa640 4 set_mon_vals no callback set 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbbf8005670 con 0x7fbc080a9920 2024-03-25T19:40:41.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:41.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.267+0000 7fbbef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:41.276 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.277+0000 7f04fdc42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:41.288 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.289+0000 7efe2da0e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:41.337 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.338+0000 7fbc0d332640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d573c566-53b7-4309-b868-cd9103387d76", "id": 0}] 2024-03-25T19:40:41.337 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.338+0000 7fbc0d332640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d573c566-53b7-4309-b868-cd9103387d76", "id": 0}] 2024-03-25T19:40:41.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.338+0000 7fbc0d332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:41.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.338+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d573c566-53b7-4309-b868-cd9103387d76", "id": 0} v 0) v1 -- 0x7fbc0804f730 con 0x7fbc080a9920 2024-03-25T19:40:41.392 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.392+0000 7f04fdc42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:41.404 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.405+0000 7efe2da0e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:41.461 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.461+0000 7f04fdc42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:41.499 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.484+0000 7efe2da0e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:41.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbbef7fe640 1 -- 172.21.15.193:0/651030988 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d573c566-53b7-4309-b868-cd9103387d76", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 4286134862 0 732707515) 0x7fbbf80058b0 con 0x7fbc080a9920 2024-03-25T19:40:41.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbbef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d573c566-53b7-4309-b868-cd9103387d76", "id": 0}] 2024-03-25T19:40:41.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbbef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:41.507 INFO:teuthology.orchestra.run.smithi193.stdout:0 2024-03-25T19:40:41.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbc0d332640 10 librados: watch_flush enter 2024-03-25T19:40:41.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbc0d332640 10 librados: watch_flush exit 2024-03-25T19:40:41.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbc0d332640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-25T19:40:41.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbc0d332640 10 client.4120.objecter successfully canceled tick 2024-03-25T19:40:41.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbc0d332640 10 monclient: shutdown 2024-03-25T19:40:41.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbc0d332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:41.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.507+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 >> v1:172.21.15.193:6789/0 conn(0x7fbc080a9920 legacy=0x7fbc08144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.508 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.508+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 shutdown_connections 2024-03-25T19:40:41.509 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.508+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 >> 172.21.15.193:0/651030988 conn(0x7fbc080b7f40 msgr2=0x7fbc08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:41.509 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.508+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 shutdown_connections 2024-03-25T19:40:41.509 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.508+0000 7fbc0d332640 1 -- 172.21.15.193:0/651030988 wait complete. 2024-03-25T19:40:41.509 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.508+0000 7fbc0d332640 1 librados: shutdown 2024-03-25T19:40:41.509 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.508+0000 7f04fdc42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:41.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 8cc2e952-5476-4395-9470-bfb55e5fe490 1 2024-03-25T19:40:41.533 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.532+0000 7efe2da0e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:41.562 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.563+0000 7f04fdc42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:41.584 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.584+0000 7efe2da0e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:41.612 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:41.612+0000 7f04fdc42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:41.630 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:41.631+0000 7efe2da0e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:41.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.658+0000 7fc422f3d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.658+0000 7fc422f3d640 10 monclient: build_initial_monmap 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.658+0000 7fc422f3d640 10 monclient: monmap: 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:40:41.659372+0000 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:40:41.659372+0000 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:41.658 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 1 Processor -- start 2024-03-25T19:40:41.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 1 -- start start 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 10 monclient: init 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c0b4700 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c142a20 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c0a96b0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0a9d60 con 0x7fc41c142a20 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c13ff10 con 0x7fc41c0a96b0 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0aa1f0 con 0x7fc41c0b4700 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.659+0000 7fc422f3d640 10 monclient(hunting): authenticate will time out at 4849.812538s 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.660+0000 7fc4214da640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc41c0a96b0 0x7fc41c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52652/0 (socket says 172.21.15.193:52652) 2024-03-25T19:40:41.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.660+0000 7fc4214da640 1 -- 172.21.15.193:0/62938824 learned_addr learned my addr 172.21.15.193:0/62938824 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:41.662 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.662+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859567599 0 0) 0x7fc41c13ff10 con 0x7fc41c0a96b0 2024-03-25T19:40:41.662 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.662+0000 7fc41b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.662+0000 7fc41b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.662+0000 7fc41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.662+0000 7fc41b7fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.662+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc40c003520 con 0x7fc41c0a96b0 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281348285 0 0) 0x7fc41c0aa1f0 con 0x7fc41c0b4700 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 10 monclient(hunting): my global_id is 14117 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c13ff10 con 0x7fc41c0b4700 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151728193 0 0) 0x7fc41c0a9d60 con 0x7fc41c142a20 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:40:41.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.663+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c0aa1f0 con 0x7fc41c142a20 2024-03-25T19:40:41.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1950999709 0 0) 0x7fc40c003520 con 0x7fc41c0a96b0 2024-03-25T19:40:41.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 >> v1:172.21.15.71:6790/0 conn(0x7fc41c0b4700 legacy=0x7fc41c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 >> v1:172.21.15.71:6789/0 conn(0x7fc41c142a20 legacy=0x7fc41c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0b4ae0 con 0x7fc41c0a96b0 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc414001a10 con 0x7fc41c0a96b0 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc422f3d640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc422f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc41b7fe640 10 monclient: dump: 2024-03-25T19:40:41.666 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.666+0000 7fc422f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.667+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc414001b70 con 0x7fc41c0a96b0 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.667+0000 7fc41b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.667+0000 7fc41b7fe640 1 -- 172.21.15.193:0/62938824 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc414001f80 con 0x7fc41c0a96b0 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.667+0000 7fc422f3d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.667+0000 7fc422f3d640 4 set_mon_vals no callback set 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.667+0000 7fc41b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.667+0000 7fc41b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.667+0000 7fc41b7fe640 10 monclient: dump: 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.669 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.668+0000 7fc422f3d640 10 monclient: shutdown 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.668+0000 7fc422f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.668+0000 7fc422f3d640 1 -- 172.21.15.193:0/62938824 >> v1:172.21.15.193:6789/0 conn(0x7fc41c0a96b0 legacy=0x7fc41c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.668+0000 7fc422f3d640 1 -- 172.21.15.193:0/62938824 shutdown_connections 2024-03-25T19:40:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.668+0000 7fc422f3d640 1 -- 172.21.15.193:0/62938824 >> 172.21.15.193:0/62938824 conn(0x7fc41c0668e0 msgr2=0x7fc41c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.668+0000 7fc422f3d640 1 -- 172.21.15.193:0/62938824 shutdown_connections 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.668+0000 7fc422f3d640 1 -- 172.21.15.193:0/62938824 wait complete. 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 10 monclient: build_initial_monmap 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 10 monclient: monmap: 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.671 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 1 librados: starting msgr at 2024-03-25T19:40:41.672 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 1 librados: starting objecter 2024-03-25T19:40:41.672 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 1 Processor -- start 2024-03-25T19:40:41.672 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 1 -- start start 2024-03-25T19:40:41.672 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 1 librados: setting wanted keys 2024-03-25T19:40:41.672 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 1 librados: calling monclient init 2024-03-25T19:40:41.672 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.669+0000 7fc422f3d640 10 monclient: init 2024-03-25T19:40:41.674 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:41.674 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c0a96b0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c0b4700 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c142a20 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c13c830 con 0x7fc41c0a96b0 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c133a90 con 0x7fc41c142a20 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0ac8f0 con 0x7fc41c0b4700 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.670+0000 7fc422f3d640 10 monclient(hunting): authenticate will time out at 4849.823107s 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.671+0000 7fc4214da640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc41c142a20 0x7fc41c133380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52668/0 (socket says 172.21.15.193:52668) 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.671+0000 7fc4214da640 1 -- 172.21.15.193:0/1211668904 learned_addr learned my addr 172.21.15.193:0/1211668904 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.671+0000 7fc4197fa640 10 client.?.objecter ms_handle_connect 0x7fc41c142a20 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.671+0000 7fc4197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.671+0000 7fc4197fa640 10 client.?.objecter ms_handle_connect 0x7fc41c0b4700 2024-03-25T19:40:41.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.671+0000 7fc4197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.671+0000 7fc4197fa640 10 client.?.objecter ms_handle_connect 0x7fc41c0a96b0 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.671+0000 7fc4197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.673+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820458060 0 0) 0x7fc41c0ac8f0 con 0x7fc41c0b4700 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.673+0000 7fc4197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.673+0000 7fc4197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.673+0000 7fc4197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.673+0000 7fc4197fa640 10 monclient(hunting): my global_id is 14120 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.673+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc400004eb0 con 0x7fc41c0b4700 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565409196 0 0) 0x7fc41c133a90 con 0x7fc41c142a20 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:40:41.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c0ac8f0 con 0x7fc41c142a20 2024-03-25T19:40:41.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714855148 0 0) 0x7fc41c13c830 con 0x7fc41c0a96b0 2024-03-25T19:40:41.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:41.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:41.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:41.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:40:41.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.674+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c133a90 con 0x7fc41c0a96b0 2024-03-25T19:40:41.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.678+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2414424011 0 0) 0x7fc400004eb0 con 0x7fc41c0b4700 2024-03-25T19:40:41.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.678+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc41c13c830 con 0x7fc41c0b4700 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.678+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc4080018f0 con 0x7fc41c0b4700 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.678+0000 7fc4197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.678+0000 7fc4197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.678+0000 7fc4197fa640 10 monclient(hunting): dump: 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.678 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.679 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.678+0000 7fc4197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:41.679 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3487063717 0 0) 0x7fc41c0ac8f0 con 0x7fc41c142a20 2024-03-25T19:40:41.679 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc400004eb0 con 0x7fc41c142a20 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc414001e80 con 0x7fc41c142a20 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 10 monclient(hunting): dump: 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.680 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4215236788 0 0) 0x7fc41c133a90 con 0x7fc41c0a96b0 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc41c0ac8f0 con 0x7fc41c0a96b0 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc410001860 con 0x7fc41c0a96b0 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.679+0000 7fc4197fa640 10 monclient(hunting): dump: 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.681 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.682 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.682 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4274797810 0 0) 0x7fc41c13c830 con 0x7fc41c0b4700 2024-03-25T19:40:41.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 >> v1:172.21.15.193:6789/0 conn(0x7fc41c142a20 legacy=0x7fc41c133380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 >> v1:172.21.15.71:6789/0 conn(0x7fc41c0a96b0 legacy=0x7fc41c138f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:41.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:41.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 1 monclient: found mon.c 2024-03-25T19:40:41.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0ac610 con 0x7fc41c0b4700 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc4197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 5 monclient: authenticate success, global_id 14120 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 10 monclient: _renew_subs 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c13ccc0 con 0x7fc41c0b4700 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 10 client.14120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 10 monclient: _renew_subs 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c1397f0 con 0x7fc41c0b4700 2024-03-25T19:40:41.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 1 librados: init done 2024-03-25T19:40:41.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.682+0000 7fc422f3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:41.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.683+0000 7fc422f3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:41.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.683+0000 7fc422f3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:40:41.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.683+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c04f730 con 0x7fc41c0b4700 2024-03-25T19:40:41.731 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc408003610 con 0x7fc41c0b4700 2024-03-25T19:40:41.731 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:41.731 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc408003920 con 0x7fc41c0b4700 2024-03-25T19:40:41.731 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 10 monclient: dump: 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc408003e50 con 0x7fc41c0b4700 2024-03-25T19:40:41.732 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc41affd640 4 set_mon_vals no callback set 2024-03-25T19:40:41.733 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 535264681 0 0) 0x7fc4080053c0 con 0x7fc41c0b4700 2024-03-25T19:40:41.733 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 10 client.14120.objecter ms_dispatch 0x7fc41c000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:40:41.733 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 3 client.14120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:40:41.733 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 3 client.14120.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:40:41.733 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.731+0000 7fc4197fa640 20 client.14120.objecter dump_active .. 0 homeless 2024-03-25T19:40:41.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.907+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc4080057a0 con 0x7fc41c0b4700 2024-03-25T19:40:41.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.907+0000 7fc4197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:41.907 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.907+0000 7fc4197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:41.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.979+0000 7fc422f3d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8cc2e952-5476-4395-9470-bfb55e5fe490", "id": 1}] 2024-03-25T19:40:41.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.979+0000 7fc422f3d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8cc2e952-5476-4395-9470-bfb55e5fe490", "id": 1}] 2024-03-25T19:40:41.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.979+0000 7fc422f3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:40:41.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:41.979+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd new", "uuid": "8cc2e952-5476-4395-9470-bfb55e5fe490", "id": 1} v 0) v1 -- 0x7fc41c0670b0 con 0x7fc41c0b4700 2024-03-25T19:40:42.017 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.017+0000 7f04fdc42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:40:42.031 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.031+0000 7efe2da0e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:40:42.120 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.120+0000 7fc4197fa640 1 -- 172.21.15.193:0/1211668904 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8cc2e952-5476-4395-9470-bfb55e5fe490", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1051259406 0 940483276) 0x7fc408005a10 con 0x7fc41c0b4700 2024-03-25T19:40:42.120 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.120+0000 7fc4197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8cc2e952-5476-4395-9470-bfb55e5fe490", "id": 1}] 2024-03-25T19:40:42.120 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.120+0000 7fc4197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:42.120 INFO:teuthology.orchestra.run.smithi193.stdout:1 2024-03-25T19:40:42.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 10 librados: watch_flush enter 2024-03-25T19:40:42.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 10 librados: watch_flush exit 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 20 client.14120.objecter shutdown clearing up homeless session... 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 10 client.14120.objecter successfully canceled tick 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 10 monclient: shutdown 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 >> v1:172.21.15.71:6790/0 conn(0x7fc41c0b4700 legacy=0x7fc41c132c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 shutdown_connections 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 >> 172.21.15.193:0/1211668904 conn(0x7fc41c0668e0 msgr2=0x7fc41c13e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 shutdown_connections 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.121+0000 7fc422f3d640 1 -- 172.21.15.193:0/1211668904 wait complete. 2024-03-25T19:40:42.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.122+0000 7fc422f3d640 1 librados: shutdown 2024-03-25T19:40:42.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 148f13c2-1acc-4900-9eaf-a56547ce9bb6 2 2024-03-25T19:40:42.154 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.155+0000 7f04fdc42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:40:42.192 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.193+0000 7efe2da0e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.253+0000 7f4cb9b32640 10 monclient: get_monmap_and_config 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.253+0000 7f4cb9b32640 10 monclient: build_initial_monmap 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.253+0000 7f4cb9b32640 10 monclient: monmap: 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:40:42.254603+0000 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:40:42.254603+0000 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:42.253 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.254 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:42.254 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:42.254 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:42.254 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.254+0000 7f4cb9b32640 1 Processor -- start 2024-03-25T19:40:42.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.254+0000 7f4cb9b32640 1 -- start start 2024-03-25T19:40:42.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.254+0000 7f4cb9b32640 10 monclient: init 2024-03-25T19:40:42.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:42.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb40b4700 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb4142a20 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb40a96b0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb40a9d60 con 0x7f4cb40a96b0 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb413ff10 con 0x7f4cb40b4700 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb40aa1f0 con 0x7f4cb4142a20 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb9b32640 10 monclient(hunting): authenticate will time out at 4850.407938s 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f4cb40b4700 0x7f4cb4142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52672/0 (socket says 172.21.15.193:52672) 2024-03-25T19:40:42.256 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.255+0000 7f4cb37fe640 1 -- 172.21.15.193:0/3485042049 learned_addr learned my addr 172.21.15.193:0/3485042049 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:42.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004107135 0 0) 0x7f4cb40a9d60 con 0x7f4cb40a96b0 2024-03-25T19:40:42.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:40:42.258 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca0003520 con 0x7f4cb40a96b0 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446554442 0 0) 0x7f4cb413ff10 con 0x7f4cb40b4700 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb40a9d60 con 0x7f4cb40b4700 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286823767 0 0) 0x7f4cb40aa1f0 con 0x7f4cb4142a20 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 10 monclient(hunting): my global_id is 14123 2024-03-25T19:40:42.259 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.258+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb413ff10 con 0x7f4cb4142a20 2024-03-25T19:40:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2086766463 0 0) 0x7f4ca0003520 con 0x7f4cb40a96b0 2024-03-25T19:40:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 >> v1:172.21.15.71:6790/0 conn(0x7f4cb4142a20 legacy=0x7f4cb413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 >> v1:172.21.15.193:6789/0 conn(0x7f4cb40b4700 legacy=0x7f4cb4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 1 monclient: found mon.noname-a 2024-03-25T19:40:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:40:42.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40b4ae0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb9b32640 5 monclient: authenticate success, global_id 4128 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb9b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4cac0019d0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb2ffd640 10 monclient: dump: 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.262 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.263 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.263 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.263 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.263 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.263 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.263 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.261+0000 7f4cb9b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:42.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.263+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cac001c20 con 0x7f4cb40a96b0 2024-03-25T19:40:42.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.263+0000 7f4cb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.263+0000 7f4cb2ffd640 1 -- 172.21.15.193:0/3485042049 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4cac001fb0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.263+0000 7f4cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.263+0000 7f4cb2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.263+0000 7f4cb2ffd640 10 monclient: dump: 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.266 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.263+0000 7f4cb9b32640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.263+0000 7f4cb9b32640 4 set_mon_vals no callback set 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.264+0000 7f4cb9b32640 10 monclient: shutdown 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.264+0000 7f4cb9b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.264+0000 7f4cb9b32640 1 -- 172.21.15.193:0/3485042049 >> v1:172.21.15.71:6789/0 conn(0x7f4cb40a96b0 legacy=0x7f4cb413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.264+0000 7f4cb9b32640 1 -- 172.21.15.193:0/3485042049 shutdown_connections 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.264+0000 7f4cb9b32640 1 -- 172.21.15.193:0/3485042049 >> 172.21.15.193:0/3485042049 conn(0x7f4cb40668e0 msgr2=0x7f4cb4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.264+0000 7f4cb9b32640 1 -- 172.21.15.193:0/3485042049 shutdown_connections 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.264+0000 7f4cb9b32640 1 -- 172.21.15.193:0/3485042049 wait complete. 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 10 monclient: build_initial_monmap 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 10 monclient: monmap: 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:42.267 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 1 librados: starting msgr at 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 1 librados: starting objecter 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 1 Processor -- start 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 1 -- start start 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 1 librados: setting wanted keys 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 1 librados: calling monclient init 2024-03-25T19:40:42.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.265+0000 7f4cb9b32640 10 monclient: init 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb40a96b0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb40b4700 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb4142a20 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb4136830 con 0x7f4cb40b4700 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb412dbc0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb41362a0 con 0x7f4cb4142a20 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb9b32640 10 monclient(hunting): authenticate will time out at 4850.418808s 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f4cb40a96b0 0x7f4cb412cda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52686/0 (socket says 172.21.15.193:52686) 2024-03-25T19:40:42.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb37fe640 1 -- 172.21.15.193:0/4217236857 learned_addr learned my addr 172.21.15.193:0/4217236857 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:42.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb0ff9640 10 client.?.objecter ms_handle_connect 0x7f4cb40a96b0 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb0ff9640 10 client.?.objecter ms_handle_connect 0x7f4cb4142a20 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.266+0000 7f4cb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.267+0000 7f4cb0ff9640 10 client.?.objecter ms_handle_connect 0x7f4cb40b4700 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.267+0000 7f4cb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442463862 0 0) 0x7f4cb412dbc0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c94004eb0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395685766 0 0) 0x7f4cb4136830 con 0x7f4cb40b4700 2024-03-25T19:40:42.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.268+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb412dbc0 con 0x7f4cb40b4700 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.269+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314164868 0 0) 0x7f4cb41362a0 con 0x7f4cb4142a20 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.269+0000 7f4cb0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.269+0000 7f4cb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.269+0000 7f4cb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.269+0000 7f4cb0ff9640 10 monclient(hunting): my global_id is 14126 2024-03-25T19:40:42.272 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.269+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb4136830 con 0x7f4cb4142a20 2024-03-25T19:40:42.273 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1541584712 0 0) 0x7f4c94004eb0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.273 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cb41362a0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.273 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4ca4001970 con 0x7f4cb40a96b0 2024-03-25T19:40:42.273 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.273 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:42.273 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1847794036 0 0) 0x7f4cb412dbc0 con 0x7f4cb40b4700 2024-03-25T19:40:42.274 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c94004eb0 con 0x7f4cb40b4700 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4c9c001870 con 0x7f4cb40b4700 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.273+0000 7f4cb0ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.275 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.277 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.275+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2747553506 0 0) 0x7f4cb4136830 con 0x7f4cb4142a20 2024-03-25T19:40:42.277 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cb412dbc0 con 0x7f4cb4142a20 2024-03-25T19:40:42.277 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4cac001d90 con 0x7f4cb4142a20 2024-03-25T19:40:42.277 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.277 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 667821229 0 0) 0x7f4cb41362a0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 >> v1:172.21.15.71:6790/0 conn(0x7f4cb4142a20 legacy=0x7f4cb412d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.278 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 >> v1:172.21.15.71:6789/0 conn(0x7f4cb40b4700 legacy=0x7f4cb4133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 1 monclient: found mon.b 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb4136f60 con 0x7f4cb40a96b0 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 10 monclient: _renew_subs 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb4137240 con 0x7f4cb40a96b0 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:42.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 10 monclient: _renew_subs 2024-03-25T19:40:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb41375d0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.276+0000 7f4cb9b32640 1 librados: init done 2024-03-25T19:40:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.277+0000 7f4cb9b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.277+0000 7f4cb9b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.277+0000 7f4cb9b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.277+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb404f730 con 0x7f4cb40a96b0 2024-03-25T19:40:42.286 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.287+0000 7f04fdc42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:40:42.306 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.307+0000 7efe2da0e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:40:42.331 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.332+0000 7f04fdc42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:40:42.351 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.351+0000 7efe2da0e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:40:42.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ca40047e0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:42.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4ca4004af0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 10 monclient: dump: 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4ca4004fe0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.374 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb27fc640 4 set_mon_vals no callback set 2024-03-25T19:40:42.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 186589543 0 0) 0x7f4ca40065b0 con 0x7f4cb40a96b0 2024-03-25T19:40:42.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 10 client.4132.objecter ms_dispatch 0x7f4cb4000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:40:42.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:40:42.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:40:42.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T19:40:42.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4ca4006990 con 0x7f4cb40a96b0 2024-03-25T19:40:42.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.375 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.373+0000 7f4cb0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:42.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.445+0000 7f4cb9b32640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "148f13c2-1acc-4900-9eaf-a56547ce9bb6", "id": 2}] 2024-03-25T19:40:42.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.445+0000 7f4cb9b32640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "148f13c2-1acc-4900-9eaf-a56547ce9bb6", "id": 2}] 2024-03-25T19:40:42.445 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.445+0000 7f4cb9b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:42.445 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.445+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "148f13c2-1acc-4900-9eaf-a56547ce9bb6", "id": 2} v 0) v1 -- 0x7f4cb413cc20 con 0x7f4cb40a96b0 2024-03-25T19:40:42.445 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.446+0000 7f04fdc42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:42.461 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.461+0000 7efe2da0e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:42.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.505+0000 7f4cb0ff9640 1 -- 172.21.15.193:0/4217236857 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "148f13c2-1acc-4900-9eaf-a56547ce9bb6", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 260280591 0 216599125) 0x7f4ca4006260 con 0x7f4cb40a96b0 2024-03-25T19:40:42.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.505+0000 7f4cb0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "148f13c2-1acc-4900-9eaf-a56547ce9bb6", "id": 2}] 2024-03-25T19:40:42.504 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.505+0000 7f4cb0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:42.505 INFO:teuthology.orchestra.run.smithi193.stdout:2 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 10 librados: watch_flush enter 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 10 librados: watch_flush exit 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 10 client.4132.objecter successfully canceled tick 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 10 monclient: shutdown 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 >> v1:172.21.15.193:6789/0 conn(0x7f4cb40a96b0 legacy=0x7f4cb412cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 shutdown_connections 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 >> 172.21.15.193:0/4217236857 conn(0x7f4cb40668e0 msgr2=0x7f4cb41416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:42.506 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 shutdown_connections 2024-03-25T19:40:42.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 1 -- 172.21.15.193:0/4217236857 wait complete. 2024-03-25T19:40:42.507 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.506+0000 7f4cb9b32640 1 librados: shutdown 2024-03-25T19:40:42.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375 3 2024-03-25T19:40:42.545 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.546+0000 7f04fdc42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:40:42.577 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.577+0000 7efe2da0e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:40:42.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.616+0000 7f49e9474640 10 monclient: get_monmap_and_config 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.616+0000 7f49e9474640 10 monclient: build_initial_monmap 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.616+0000 7f49e9474640 10 monclient: monmap: 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:40:42.617378+0000 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:40:42.617378+0000 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:42.616 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 1 Processor -- start 2024-03-25T19:40:42.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 1 -- start start 2024-03-25T19:40:42.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 10 monclient: init 2024-03-25T19:40:42.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:42.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:42.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 10 monclient(hunting): picked mon.noname-b con 0x7f49e40b4700 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:42.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 10 monclient(hunting): picked mon.noname-a con 0x7f49e4142a20 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:42.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 10 monclient(hunting): picked mon.noname-c con 0x7f49e40a96b0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:42.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40a9d60 con 0x7f49e4142a20 2024-03-25T19:40:42.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e413ff10 con 0x7f49e40a96b0 2024-03-25T19:40:42.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40aa1f0 con 0x7f49e40b4700 2024-03-25T19:40:42.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:42.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.617+0000 7f49e9474640 10 monclient(hunting): authenticate will time out at 4850.770424s 2024-03-25T19:40:42.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.618+0000 7f49e37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f49e40a96b0 0x7f49e413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52700/0 (socket says 172.21.15.193:52700) 2024-03-25T19:40:42.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.618+0000 7f49e37fe640 1 -- 172.21.15.193:0/3142192735 learned_addr learned my addr 172.21.15.193:0/3142192735 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:42.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.620+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388610874 0 0) 0x7f49e413ff10 con 0x7f49e40a96b0 2024-03-25T19:40:42.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.620+0000 7f49e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.620+0000 7f49e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.620+0000 7f49e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.620+0000 7f49e1ffb640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:40:42.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.620+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49cc003520 con 0x7f49e40a96b0 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026977638 0 0) 0x7f49e40a9d60 con 0x7f49e4142a20 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e413ff10 con 0x7f49e4142a20 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655089383 0 0) 0x7f49e40aa1f0 con 0x7f49e40b4700 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 10 monclient(hunting): my global_id is 14129 2024-03-25T19:40:42.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.621+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e40a9d60 con 0x7f49e40b4700 2024-03-25T19:40:42.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 930244064 0 0) 0x7f49cc003520 con 0x7f49e40a96b0 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 >> v1:172.21.15.71:6790/0 conn(0x7f49e40b4700 legacy=0x7f49e4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 >> v1:172.21.15.71:6789/0 conn(0x7f49e4142a20 legacy=0x7f49e413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 1 monclient: found mon.noname-c 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e40b4ae0 con 0x7f49e40a96b0 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f49dc001a10 con 0x7f49e40a96b0 2024-03-25T19:40:42.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e9474640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e9474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e1ffb640 10 monclient: dump: 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.625 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.623+0000 7f49e9474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49dc001b70 con 0x7f49e40a96b0 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e9474640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e1ffb640 1 -- 172.21.15.193:0/3142192735 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f49dc001f80 con 0x7f49e40a96b0 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e9474640 4 set_mon_vals no callback set 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e1ffb640 10 monclient: dump: 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.629 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e9474640 10 monclient: shutdown 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e9474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.625+0000 7f49e9474640 1 -- 172.21.15.193:0/3142192735 >> v1:172.21.15.193:6789/0 conn(0x7f49e40a96b0 legacy=0x7f49e413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.626+0000 7f49e9474640 1 -- 172.21.15.193:0/3142192735 shutdown_connections 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.626+0000 7f49e9474640 1 -- 172.21.15.193:0/3142192735 >> 172.21.15.193:0/3142192735 conn(0x7f49e40668e0 msgr2=0x7f49e4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.626+0000 7f49e9474640 1 -- 172.21.15.193:0/3142192735 shutdown_connections 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.626+0000 7f49e9474640 1 -- 172.21.15.193:0/3142192735 wait complete. 2024-03-25T19:40:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.626+0000 7f49e9474640 10 monclient: build_initial_monmap 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.626+0000 7f49e9474640 10 monclient: monmap: 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.626+0000 7f49e9474640 1 librados: starting msgr at 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.626+0000 7f49e9474640 1 librados: starting objecter 2024-03-25T19:40:42.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 1 Processor -- start 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 1 -- start start 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 1 librados: setting wanted keys 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 1 librados: calling monclient init 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 10 monclient: init 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 10 monclient(hunting): picked mon.noname-a con 0x7f49e40a96b0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 10 monclient(hunting): picked mon.noname-c con 0x7f49e40b4700 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 10 monclient(hunting): picked mon.noname-b con 0x7f49e4142a20 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40ab970 con 0x7f49e40a96b0 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e413c050 con 0x7f49e4142a20 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e41d81c0 con 0x7f49e40b4700 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.627+0000 7f49e9474640 10 monclient(hunting): authenticate will time out at 4850.780521s 2024-03-25T19:40:42.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.628+0000 7f49e37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f49e4142a20 0x7f49e41d4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52702/0 (socket says 172.21.15.193:52702) 2024-03-25T19:40:42.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.628+0000 7f49e37fe640 1 -- 172.21.15.193:0/4256524591 learned_addr learned my addr 172.21.15.193:0/4256524591 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:42.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.628+0000 7f49cb7fe640 10 client.?.objecter ms_handle_connect 0x7f49e4142a20 2024-03-25T19:40:42.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.628+0000 7f49cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.628+0000 7f49cb7fe640 10 client.?.objecter ms_handle_connect 0x7f49e40a96b0 2024-03-25T19:40:42.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.628+0000 7f49cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.628+0000 7f49cb7fe640 10 client.?.objecter ms_handle_connect 0x7f49e40b4700 2024-03-25T19:40:42.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.628+0000 7f49cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:42.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.630+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863404980 0 0) 0x7f49e40ab970 con 0x7f49e40a96b0 2024-03-25T19:40:42.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.630+0000 7f49cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.630+0000 7f49cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.630+0000 7f49cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.630+0000 7f49cb7fe640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.630+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d0004eb0 con 0x7f49e40a96b0 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546495181 0 0) 0x7f49e413c050 con 0x7f49e4142a20 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e40ab970 con 0x7f49e4142a20 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449051989 0 0) 0x7f49e41d81c0 con 0x7f49e40b4700 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 10 monclient(hunting): my global_id is 14132 2024-03-25T19:40:42.635 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.631+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e413c050 con 0x7f49e40b4700 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.635+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1300124860 0 0) 0x7f49d0004eb0 con 0x7f49e40a96b0 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.635+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e41d81c0 con 0x7f49e40a96b0 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.635+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f49d40018f0 con 0x7f49e40a96b0 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.635+0000 7f49cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.635+0000 7f49cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.635+0000 7f49cb7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.637 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.637 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.637 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.637 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.637 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.637 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.637 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.635+0000 7f49cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.637+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3864654609 0 0) 0x7f49e413c050 con 0x7f49e40b4700 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.637+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49d0004eb0 con 0x7f49e40b4700 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.637+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f49d80018d0 con 0x7f49e40b4700 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.637+0000 7f49cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.637+0000 7f49cb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.637+0000 7f49cb7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.638 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.638+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3015043442 0 0) 0x7f49e40ab970 con 0x7f49e4142a20 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.638+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e413c050 con 0x7f49e4142a20 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.638+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f49dc001e80 con 0x7f49e4142a20 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.638+0000 7f49cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.638+0000 7f49cb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.638+0000 7f49cb7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:42.639 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.640 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 938396723 0 0) 0x7f49e41d81c0 con 0x7f49e40a96b0 2024-03-25T19:40:42.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 >> v1:172.21.15.71:6790/0 conn(0x7f49e40b4700 legacy=0x7f49e41d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 >> v1:172.21.15.193:6789/0 conn(0x7f49e4142a20 legacy=0x7f49e41d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 1 monclient: found mon.a 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e41d7590 con 0x7f49e40a96b0 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 5 monclient: authenticate success, global_id 4137 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 10 monclient: _renew_subs 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:42.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e41d4e90 con 0x7f49e40a96b0 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 10 monclient: _renew_subs 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e413ae10 con 0x7f49e40a96b0 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.640+0000 7f49e9474640 1 librados: init done 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.641+0000 7f49e9474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.641+0000 7f49e9474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.641+0000 7f49e9474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:42.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.641+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e41d7720 con 0x7f49e40a96b0 2024-03-25T19:40:42.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49d4003610 con 0x7f49e40a96b0 2024-03-25T19:40:42.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:42.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f49d4003920 con 0x7f49e40a96b0 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 10 monclient: dump: 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:42.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f49d4003e10 con 0x7f49e40a96b0 2024-03-25T19:40:42.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49e17fa640 4 set_mon_vals no callback set 2024-03-25T19:40:42.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3148274964 0 0) 0x7f49d4005410 con 0x7f49e40a96b0 2024-03-25T19:40:42.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 10 client.4137.objecter ms_dispatch 0x7f49e4000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:40:42.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:40:42.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:40:42.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.645+0000 7f49cb7fe640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T19:40:42.710 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.711+0000 7f04fdc42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:42.741 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.742+0000 7efe2da0e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:42.752 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.753+0000 7f04fdc42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:42.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.755+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f49d4005090 con 0x7f49e40a96b0 2024-03-25T19:40:42.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.755+0000 7f49cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:42.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.755+0000 7f49cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:42.785 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.786+0000 7efe2da0e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:42.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.828+0000 7f49e9474640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375", "id": 3}] 2024-03-25T19:40:42.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.828+0000 7f49e9474640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375", "id": 3}] 2024-03-25T19:40:42.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.828+0000 7f49e9474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:42.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.828+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375", "id": 3} v 0) v1 -- 0x7f49e41d7380 con 0x7f49e40a96b0 2024-03-25T19:40:42.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.863+0000 7f49cb7fe640 1 -- 172.21.15.193:0/4256524591 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1772736665 0 525046306) 0x7f49d4005620 con 0x7f49e40a96b0 2024-03-25T19:40:42.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.863+0000 7f49cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375", "id": 3}] 2024-03-25T19:40:42.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.863+0000 7f49cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:42.863 INFO:teuthology.orchestra.run.smithi193.stdout:3 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 10 librados: watch_flush enter 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 10 librados: watch_flush exit 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 10 client.4137.objecter successfully canceled tick 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 10 monclient: shutdown 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 >> v1:172.21.15.71:6789/0 conn(0x7f49e40a96b0 legacy=0x7f49e413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 shutdown_connections 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 >> 172.21.15.193:0/4256524591 conn(0x7f49e40668e0 msgr2=0x7f49e413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 shutdown_connections 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.864+0000 7f49e9474640 1 -- 172.21.15.193:0/4256524591 wait complete. 2024-03-25T19:40:42.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:42.865+0000 7f49e9474640 1 librados: shutdown 2024-03-25T19:40:42.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 504df1e2-17c5-4070-bcf7-ecce4030c19f 4 2024-03-25T19:40:42.934 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:42.934+0000 7f04fdc42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:40:42.954 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:42.955+0000 7efe2da0e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.017+0000 7f394eafb640 10 monclient: get_monmap_and_config 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.017+0000 7f394eafb640 10 monclient: build_initial_monmap 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.017+0000 7f394eafb640 10 monclient: monmap: 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:40:43.018346+0000 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:40:43.018346+0000 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:43.018 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:43.018 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 1 Processor -- start 2024-03-25T19:40:43.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 1 -- start start 2024-03-25T19:40:43.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 10 monclient: init 2024-03-25T19:40:43.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:43.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:43.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 10 monclient(hunting): picked mon.noname-c con 0x7f39480b4700 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:43.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 10 monclient(hunting): picked mon.noname-b con 0x7f3948142a20 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:43.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 10 monclient(hunting): picked mon.noname-a con 0x7f39480a96b0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:43.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480a9d60 con 0x7f39480a96b0 2024-03-25T19:40:43.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394813ff10 con 0x7f39480b4700 2024-03-25T19:40:43.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480aa1f0 con 0x7f3948142a20 2024-03-25T19:40:43.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.018+0000 7f394eafb640 10 monclient(hunting): authenticate will time out at 4851.171492s 2024-03-25T19:40:43.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.019+0000 7f394c897640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f39480b4700 0x7f3948142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52704/0 (socket says 172.21.15.193:52704) 2024-03-25T19:40:43.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.019+0000 7f394c897640 1 -- 172.21.15.193:0/1266900160 learned_addr learned my addr 172.21.15.193:0/1266900160 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:43.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.021+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45808362 0 0) 0x7f394813ff10 con 0x7f39480b4700 2024-03-25T19:40:43.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.021+0000 7f39477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.021+0000 7f39477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.021+0000 7f39477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.021+0000 7f39477fe640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:40:43.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.021+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3938003520 con 0x7f39480b4700 2024-03-25T19:40:43.021 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.022+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637904493 0 0) 0x7f39480aa1f0 con 0x7f3948142a20 2024-03-25T19:40:43.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.022+0000 7f39477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.022+0000 7f39477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.022+0000 7f39477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.022+0000 7f39477fe640 10 monclient(hunting): my global_id is 14135 2024-03-25T19:40:43.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.022+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394813ff10 con 0x7f3948142a20 2024-03-25T19:40:43.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.023+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781872733 0 0) 0x7f39480a9d60 con 0x7f39480a96b0 2024-03-25T19:40:43.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.023+0000 7f39477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.023+0000 7f39477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.023+0000 7f39477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.023+0000 7f39477fe640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:40:43.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.023+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39480aa1f0 con 0x7f39480a96b0 2024-03-25T19:40:43.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2552001629 0 0) 0x7f3938003520 con 0x7f39480b4700 2024-03-25T19:40:43.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 >> v1:172.21.15.71:6790/0 conn(0x7f3948142a20 legacy=0x7f394813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 >> v1:172.21.15.71:6789/0 conn(0x7f39480a96b0 legacy=0x7f394813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:43.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:43.024 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39480b4ae0 con 0x7f39480b4700 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f393c001690 con 0x7f39480b4700 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f394eafb640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f394eafb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f39477fe640 10 monclient: dump: 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.026 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.026 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.026 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.026 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.026 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.026 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.026 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.026 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.024+0000 7f394eafb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:43.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.026+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f393c0017f0 con 0x7f39480b4700 2024-03-25T19:40:43.027 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.026+0000 7f39477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.026+0000 7f39477fe640 1 -- 172.21.15.193:0/1266900160 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f393c001c00 con 0x7f39480b4700 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.026+0000 7f39477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.026+0000 7f39477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.026+0000 7f39477fe640 10 monclient: dump: 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.028 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.026+0000 7f394eafb640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.026+0000 7f394eafb640 4 set_mon_vals no callback set 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.027+0000 7f394eafb640 10 monclient: shutdown 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.027+0000 7f394eafb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.027+0000 7f394eafb640 1 -- 172.21.15.193:0/1266900160 >> v1:172.21.15.193:6789/0 conn(0x7f39480b4700 legacy=0x7f3948142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.027+0000 7f394eafb640 1 -- 172.21.15.193:0/1266900160 shutdown_connections 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.027+0000 7f394eafb640 1 -- 172.21.15.193:0/1266900160 >> 172.21.15.193:0/1266900160 conn(0x7f39480668e0 msgr2=0x7f3948066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.027+0000 7f394eafb640 1 -- 172.21.15.193:0/1266900160 shutdown_connections 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.027+0000 7f394eafb640 1 -- 172.21.15.193:0/1266900160 wait complete. 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.028+0000 7f394eafb640 10 monclient: build_initial_monmap 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.028+0000 7f394eafb640 10 monclient: monmap: 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T19:40:43.029 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T19:40:43.030 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:43.030 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T19:40:43.030 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:43.030 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:43.030 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:43.030 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.028+0000 7f394eafb640 1 librados: starting msgr at 2024-03-25T19:40:43.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.028+0000 7f394eafb640 1 librados: starting objecter 2024-03-25T19:40:43.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.028+0000 7f394eafb640 1 Processor -- start 2024-03-25T19:40:43.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.029+0000 7f394eafb640 1 -- start start 2024-03-25T19:40:43.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.029+0000 7f394eafb640 1 librados: setting wanted keys 2024-03-25T19:40:43.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.029+0000 7f394eafb640 1 librados: calling monclient init 2024-03-25T19:40:43.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.029+0000 7f394eafb640 10 monclient: init 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 10 monclient(hunting): picked mon.noname-b con 0x7f39480a96b0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 10 monclient(hunting): picked mon.noname-c con 0x7f39480b4700 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 10 monclient(hunting): picked mon.noname-a con 0x7f3948142a20 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480ae7c0 con 0x7f3948142a20 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480ad8e0 con 0x7f39480a96b0 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394813a1c0 con 0x7f39480b4700 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394eafb640 10 monclient(hunting): authenticate will time out at 4851.182836s 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394c897640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f39480a96b0 0x7f39480ad1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52716/0 (socket says 172.21.15.193:52716) 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f394c897640 1 -- 172.21.15.193:0/636732964 learned_addr learned my addr 172.21.15.193:0/636732964 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f39457fa640 10 client.?.objecter ms_handle_connect 0x7f39480a96b0 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.030+0000 7f39457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.031+0000 7f39457fa640 10 client.?.objecter ms_handle_connect 0x7f39480b4700 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.031+0000 7f39457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.031+0000 7f39457fa640 10 client.?.objecter ms_handle_connect 0x7f3948142a20 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.031+0000 7f39457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.032+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734506636 0 0) 0x7f39480ad8e0 con 0x7f39480a96b0 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.032+0000 7f39457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.032+0000 7f39457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.032+0000 7f39457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.032+0000 7f39457fa640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:40:43.034 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.032+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f392c004eb0 con 0x7f39480a96b0 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.033+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706152663 0 0) 0x7f394813a1c0 con 0x7f39480b4700 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.033+0000 7f39457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.033+0000 7f39457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.033+0000 7f39457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.033+0000 7f39457fa640 10 monclient(hunting): my global_id is 14138 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.033+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39480ad8e0 con 0x7f39480b4700 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.034+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648302011 0 0) 0x7f39480ae7c0 con 0x7f3948142a20 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.034+0000 7f39457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.034+0000 7f39457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.034+0000 7f39457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.034+0000 7f39457fa640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:40:43.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.034+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394813a1c0 con 0x7f3948142a20 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.037+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2740017052 0 0) 0x7f392c004eb0 con 0x7f39480a96b0 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.037+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39480ae7c0 con 0x7f39480a96b0 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.037+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f393c001bc0 con 0x7f39480a96b0 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.037+0000 7f39457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.037+0000 7f39457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.037+0000 7f39457fa640 10 monclient(hunting): dump: 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.037 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.038 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.038 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.038 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.038 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.038 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.038 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.038 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.037+0000 7f39457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:43.039 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.039+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3696517663 0 0) 0x7f39480ad8e0 con 0x7f39480b4700 2024-03-25T19:40:43.039 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.039+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f392c004eb0 con 0x7f39480b4700 2024-03-25T19:40:43.039 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.039+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3934001970 con 0x7f39480b4700 2024-03-25T19:40:43.039 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.039+0000 7f39457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.039 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.039+0000 7f39457fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:43.039 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.039+0000 7f39457fa640 10 monclient(hunting): dump: 2024-03-25T19:40:43.039 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.039 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.040 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.040 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.040 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.040 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.040 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.040 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.040 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.040 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1377698083 0 0) 0x7f39480ae7c0 con 0x7f39480a96b0 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 >> v1:172.21.15.71:6790/0 conn(0x7f39480b4700 legacy=0x7f394813a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 >> v1:172.21.15.71:6789/0 conn(0x7f3948142a20 legacy=0x7f394813add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 1 monclient: found mon.b 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39480aeaa0 con 0x7f39480a96b0 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:43.042 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f39457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 10 monclient: _renew_subs 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39480ac440 con 0x7f39480a96b0 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 10 monclient: _renew_subs 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39481e39c0 con 0x7f39480a96b0 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.041+0000 7f394eafb640 1 librados: init done 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.042+0000 7f394eafb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.042+0000 7f394eafb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.042+0000 7f394eafb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.043 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.042+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394813a350 con 0x7f39480a96b0 2024-03-25T19:40:43.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f393c0011f0 con 0x7f39480a96b0 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f393c004a90 con 0x7f39480a96b0 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 10 monclient: dump: 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.045 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.046 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.046 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f3946ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:43.046 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f393c005000 con 0x7f39480a96b0 2024-03-25T19:40:43.046 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 537478410 0 0) 0x7f393c0066b0 con 0x7f39480a96b0 2024-03-25T19:40:43.046 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 10 client.4144.objecter ms_dispatch 0x7f3948000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:40:43.046 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:40:43.046 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:40:43.046 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.045+0000 7f39457fa640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:40:43.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.158+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f393c0062b0 con 0x7f39480a96b0 2024-03-25T19:40:43.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.158+0000 7f39457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.158+0000 7f39457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:43.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.232+0000 7f394eafb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "504df1e2-17c5-4070-bcf7-ecce4030c19f", "id": 4}] 2024-03-25T19:40:43.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.232+0000 7f394eafb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "504df1e2-17c5-4070-bcf7-ecce4030c19f", "id": 4}] 2024-03-25T19:40:43.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.232+0000 7f394eafb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.232+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "504df1e2-17c5-4070-bcf7-ecce4030c19f", "id": 4} v 0) v1 -- 0x7f394804f730 con 0x7f39480a96b0 2024-03-25T19:40:43.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.290+0000 7f39457fa640 1 -- 172.21.15.193:0/636732964 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "504df1e2-17c5-4070-bcf7-ecce4030c19f", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 798927763 0 1697012583) 0x7f393c0068c0 con 0x7f39480a96b0 2024-03-25T19:40:43.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.290+0000 7f39457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "504df1e2-17c5-4070-bcf7-ecce4030c19f", "id": 4}] 2024-03-25T19:40:43.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.290+0000 7f39457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:43.290 INFO:teuthology.orchestra.run.smithi193.stdout:4 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 10 librados: watch_flush enter 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 10 librados: watch_flush exit 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 10 monclient: shutdown 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 >> v1:172.21.15.193:6789/0 conn(0x7f39480a96b0 legacy=0x7f39480ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 shutdown_connections 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 >> 172.21.15.193:0/636732964 conn(0x7f39480668e0 msgr2=0x7f394813c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 shutdown_connections 2024-03-25T19:40:43.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 1 -- 172.21.15.193:0/636732964 wait complete. 2024-03-25T19:40:43.292 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.291+0000 7f394eafb640 1 librados: shutdown 2024-03-25T19:40:43.301 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 9a964fdb-6a47-40ef-92ac-d6ebe592e84d 5 2024-03-25T19:40:43.443 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.444+0000 7fab28fa0640 10 monclient: get_monmap_and_config 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.444+0000 7fab28fa0640 10 monclient: build_initial_monmap 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.444+0000 7fab28fa0640 10 monclient: monmap: 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:40:43.445107+0000 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:40:43.445107+0000 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:43.444 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.444+0000 7fab28fa0640 1 Processor -- start 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 1 -- start start 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 10 monclient: init 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 10 monclient(hunting): picked mon.noname-c con 0x7fab2406ca40 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 10 monclient(hunting): picked mon.noname-a con 0x7fab2413fb00 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 10 monclient(hunting): picked mon.noname-b con 0x7fab2406ae90 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2406b8d0 con 0x7fab2413fb00 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2406b3b0 con 0x7fab2406ca40 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab24066ed0 con 0x7fab2406ae90 2024-03-25T19:40:43.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.445+0000 7fab28fa0640 10 monclient(hunting): authenticate will time out at 4851.598283s 2024-03-25T19:40:43.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.446+0000 7fab237fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fab2406ca40 0x7fab2413f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52720/0 (socket says 172.21.15.193:52720) 2024-03-25T19:40:43.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.446+0000 7fab237fe640 1 -- 172.21.15.193:0/392282508 learned_addr learned my addr 172.21.15.193:0/392282508 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:43.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.448+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614260777 0 0) 0x7fab2406b3b0 con 0x7fab2406ca40 2024-03-25T19:40:43.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.448+0000 7fab227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.448+0000 7fab227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.448+0000 7fab227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.448+0000 7fab227fc640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:40:43.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.448+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0c003520 con 0x7fab2406ca40 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741080213 0 0) 0x7fab24066ed0 con 0x7fab2406ae90 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 10 monclient(hunting): my global_id is 14141 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab2406b3b0 con 0x7fab2406ae90 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051392368 0 0) 0x7fab2406b8d0 con 0x7fab2413fb00 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:40:43.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.449+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab24066ed0 con 0x7fab2413fb00 2024-03-25T19:40:43.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2610370636 0 0) 0x7fab0c003520 con 0x7fab2406ca40 2024-03-25T19:40:43.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 >> v1:172.21.15.71:6790/0 conn(0x7fab2406ae90 legacy=0x7fab2414d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 >> v1:172.21.15.71:6789/0 conn(0x7fab2413fb00 legacy=0x7fab24145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:43.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 1 monclient: found mon.noname-c 2024-03-25T19:40:43.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:43.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab240adfb0 con 0x7fab2406ca40 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fab18001690 con 0x7fab2406ca40 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab28fa0640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab28fa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab227fc640 10 monclient: dump: 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.452 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.453 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.453 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.453 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.453 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.453 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.451+0000 7fab28fa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab180017f0 con 0x7fab2406ca40 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab227fc640 1 -- 172.21.15.193:0/392282508 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fab18001c00 con 0x7fab2406ca40 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 4 set_mon_vals no callback set 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab227fc640 10 monclient: dump: 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.456 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 10 monclient: shutdown 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 1 -- 172.21.15.193:0/392282508 >> v1:172.21.15.193:6789/0 conn(0x7fab2406ca40 legacy=0x7fab2413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 1 -- 172.21.15.193:0/392282508 shutdown_connections 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 1 -- 172.21.15.193:0/392282508 >> 172.21.15.193:0/392282508 conn(0x7fab2406bcb0 msgr2=0x7fab2406c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 1 -- 172.21.15.193:0/392282508 shutdown_connections 2024-03-25T19:40:43.457 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.453+0000 7fab28fa0640 1 -- 172.21.15.193:0/392282508 wait complete. 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.454+0000 7fab28fa0640 10 monclient: build_initial_monmap 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.454+0000 7fab28fa0640 10 monclient: monmap: 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.454+0000 7fab28fa0640 1 librados: starting msgr at 2024-03-25T19:40:43.458 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.454+0000 7fab28fa0640 1 librados: starting objecter 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.454+0000 7fab28fa0640 1 Processor -- start 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 1 -- start start 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 1 librados: setting wanted keys 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 1 librados: calling monclient init 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 10 monclient: init 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 10 monclient(hunting): picked mon.noname-c con 0x7fab2406ae90 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 10 monclient(hunting): picked mon.noname-b con 0x7fab2406ca40 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 10 monclient(hunting): picked mon.noname-a con 0x7fab2413fb00 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2406fec0 con 0x7fab2413fb00 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab24142d00 con 0x7fab2406ca40 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab241427e0 con 0x7fab2406ae90 2024-03-25T19:40:43.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab28fa0640 10 monclient(hunting): authenticate will time out at 4851.608252s 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.455+0000 7fab22ffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fab2406ca40 0x7fab2413ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52734/0 (socket says 172.21.15.193:52734) 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.456+0000 7fab22ffd640 1 -- 172.21.15.193:0/580553292 learned_addr learned my addr 172.21.15.193:0/580553292 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.456+0000 7fab0bfff640 10 client.?.objecter ms_handle_connect 0x7fab2406ca40 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.456+0000 7fab0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.456+0000 7fab0bfff640 10 client.?.objecter ms_handle_connect 0x7fab2413fb00 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.456+0000 7fab0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.456+0000 7fab0bfff640 10 client.?.objecter ms_handle_connect 0x7fab2406ae90 2024-03-25T19:40:43.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.456+0000 7fab0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.457+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380659086 0 0) 0x7fab2406fec0 con 0x7fab2413fb00 2024-03-25T19:40:43.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.457+0000 7fab0bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.457+0000 7fab0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.457+0000 7fab0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.457+0000 7fab0bfff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.457+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab10004eb0 con 0x7fab2413fb00 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274156096 0 0) 0x7fab24142d00 con 0x7fab2406ca40 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab2406fec0 con 0x7fab2406ca40 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477393301 0 0) 0x7fab241427e0 con 0x7fab2406ae90 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 10 monclient(hunting): my global_id is 14144 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.458+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab24142d00 con 0x7fab2406ae90 2024-03-25T19:40:43.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3721937778 0 0) 0x7fab2406fec0 con 0x7fab2406ca40 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab241427e0 con 0x7fab2406ca40 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fab140019b0 con 0x7fab2406ca40 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 10 monclient(hunting): dump: 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.464 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 940842545 0 0) 0x7fab10004eb0 con 0x7fab2413fb00 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab2406fec0 con 0x7fab2413fb00 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fab1c001bd0 con 0x7fab2413fb00 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 10 monclient(hunting): dump: 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.465 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.462+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616584671 0 0) 0x7fab24142d00 con 0x7fab2406ae90 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.463+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab10004eb0 con 0x7fab2406ae90 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.463+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fab18001b00 con 0x7fab2406ae90 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.463+0000 7fab0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.463+0000 7fab0bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.463+0000 7fab0bfff640 10 monclient(hunting): dump: 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.466 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.467 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.467 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.467 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.467 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.467 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4046310269 0 0) 0x7fab2406fec0 con 0x7fab2413fb00 2024-03-25T19:40:43.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 >> v1:172.21.15.71:6790/0 conn(0x7fab2406ae90 legacy=0x7fab24132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 >> v1:172.21.15.193:6789/0 conn(0x7fab2406ca40 legacy=0x7fab2413ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:43.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 1 monclient: found mon.a 2024-03-25T19:40:43.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:43.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:43.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab24142970 con 0x7fab2413fb00 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 10 monclient: _finish_auth 0 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 5 monclient: authenticate success, global_id 4149 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 10 monclient: _renew_subs 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab241332c0 con 0x7fab2413fb00 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 10 monclient: _renew_subs 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab24144f50 con 0x7fab2413fb00 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.465+0000 7fab28fa0640 1 librados: init done 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.466+0000 7fab28fa0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.466+0000 7fab28fa0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.466+0000 7fab28fa0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:43.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.466+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab24142b60 con 0x7fab2413fb00 2024-03-25T19:40:43.562 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab1c0020e0 con 0x7fab2413fb00 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fab1c003970 con 0x7fab2413fb00 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 10 monclient: dump: 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.563 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.564 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.564 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fab1c003ee0 con 0x7fab2413fb00 2024-03-25T19:40:43.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab21ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:43.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 926282115 0 0) 0x7fab1c0056a0 con 0x7fab2413fb00 2024-03-25T19:40:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.563+0000 7fab0bfff640 10 client.4149.objecter ms_dispatch 0x7fab24000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:40:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.564+0000 7fab0bfff640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:40:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.564+0000 7fab0bfff640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:40:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.564+0000 7fab0bfff640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T19:40:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.564+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fab1c005120 con 0x7fab2413fb00 2024-03-25T19:40:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.564+0000 7fab0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.565 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.564+0000 7fab0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:43.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.636+0000 7fab28fa0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9a964fdb-6a47-40ef-92ac-d6ebe592e84d", "id": 5}] 2024-03-25T19:40:43.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.636+0000 7fab28fa0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9a964fdb-6a47-40ef-92ac-d6ebe592e84d", "id": 5}] 2024-03-25T19:40:43.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.636+0000 7fab28fa0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:40:43.636 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.636+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9a964fdb-6a47-40ef-92ac-d6ebe592e84d", "id": 5} v 0) v1 -- 0x7fab24142e90 con 0x7fab2413fb00 2024-03-25T19:40:43.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.664+0000 7fab0bfff640 1 -- 172.21.15.193:0/580553292 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9a964fdb-6a47-40ef-92ac-d6ebe592e84d", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2848596248 0 1988412176) 0x7fab1c005410 con 0x7fab2413fb00 2024-03-25T19:40:43.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.664+0000 7fab0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9a964fdb-6a47-40ef-92ac-d6ebe592e84d", "id": 5}] 2024-03-25T19:40:43.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.664+0000 7fab0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:43.665 INFO:teuthology.orchestra.run.smithi193.stdout:5 2024-03-25T19:40:43.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 10 librados: watch_flush enter 2024-03-25T19:40:43.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 10 librados: watch_flush exit 2024-03-25T19:40:43.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T19:40:43.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 10 client.4149.objecter successfully canceled tick 2024-03-25T19:40:43.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 10 monclient: shutdown 2024-03-25T19:40:43.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:43.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 >> v1:172.21.15.71:6789/0 conn(0x7fab2413fb00 legacy=0x7fab241e2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 shutdown_connections 2024-03-25T19:40:43.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.665+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 >> 172.21.15.193:0/580553292 conn(0x7fab2406bcb0 msgr2=0x7fab2406c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:43.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.666+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 shutdown_connections 2024-03-25T19:40:43.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.666+0000 7fab28fa0640 1 -- 172.21.15.193:0/580553292 wait complete. 2024-03-25T19:40:43.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.666+0000 7fab28fa0640 1 librados: shutdown 2024-03-25T19:40:43.676 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new d50af945-a618-4993-97bc-b141ea9f3237 6 2024-03-25T19:40:43.775 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:43.776+0000 7efe2da0e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:40:43.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.776+0000 7f6e2af4d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:43.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.776+0000 7f6e2af4d640 10 monclient: build_initial_monmap 2024-03-25T19:40:43.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.776+0000 7f6e2af4d640 10 monclient: monmap: 2024-03-25T19:40:43.776 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:43.776 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.776 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:40:43.777110+0000 2024-03-25T19:40:43.776 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:40:43.777110+0000 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.776+0000 7f6e2af4d640 1 Processor -- start 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 1 -- start start 2024-03-25T19:40:43.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 10 monclient: init 2024-03-25T19:40:43.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:43.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:43.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e240a9920 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:43.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e240b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:43.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e24146450 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:43.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2406b100 con 0x7f6e24146450 2024-03-25T19:40:43.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e240b2940 con 0x7f6e240b45a0 2024-03-25T19:40:43.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2413fec0 con 0x7f6e240a9920 2024-03-25T19:40:43.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.777+0000 7f6e2af4d640 10 monclient(hunting): authenticate will time out at 4851.930310s 2024-03-25T19:40:43.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.778+0000 7f6e23fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f6e240b45a0 0x7f6e24145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52750/0 (socket says 172.21.15.193:52750) 2024-03-25T19:40:43.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.778+0000 7f6e23fff640 1 -- 172.21.15.193:0/2462815750 learned_addr learned my addr 172.21.15.193:0/2462815750 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:43.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506437621 0 0) 0x7f6e2406b100 con 0x7f6e24146450 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e14003520 con 0x7f6e24146450 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093903497 0 0) 0x7f6e240b2940 con 0x7f6e240b45a0 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2406b100 con 0x7f6e240b45a0 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379705169 0 0) 0x7f6e2413fec0 con 0x7f6e240a9920 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 10 monclient(hunting): my global_id is 14147 2024-03-25T19:40:43.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.781+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e240b2940 con 0x7f6e240a9920 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148261449 0 0) 0x7f6e2406b100 con 0x7f6e240b45a0 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 >> v1:172.21.15.71:6790/0 conn(0x7f6e240a9920 legacy=0x7f6e2413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 >> v1:172.21.15.71:6789/0 conn(0x7f6e24146450 legacy=0x7f6e2414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24140050 con 0x7f6e240b45a0 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6e10001670 con 0x7f6e240b45a0 2024-03-25T19:40:43.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e237fe640 10 monclient: dump: 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e2af4d640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:40:43.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.784+0000 7f6e2af4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:43.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.785+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e100017d0 con 0x7f6e240b45a0 2024-03-25T19:40:43.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.785+0000 7f6e237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:43.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.785+0000 7f6e237fe640 1 -- 172.21.15.193:0/2462815750 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6e10001a40 con 0x7f6e240b45a0 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.785+0000 7f6e237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.785+0000 7f6e237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.785+0000 7f6e237fe640 10 monclient: dump: 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.785+0000 7f6e2af4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.785+0000 7f6e2af4d640 4 set_mon_vals no callback set 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.786+0000 7f6e2af4d640 10 monclient: shutdown 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.786+0000 7f6e2af4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.786+0000 7f6e2af4d640 1 -- 172.21.15.193:0/2462815750 >> v1:172.21.15.193:6789/0 conn(0x7f6e240b45a0 legacy=0x7f6e24145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.786+0000 7f6e2af4d640 1 -- 172.21.15.193:0/2462815750 shutdown_connections 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.786+0000 7f6e2af4d640 1 -- 172.21.15.193:0/2462815750 >> 172.21.15.193:0/2462815750 conn(0x7f6e240b7f40 msgr2=0x7f6e240b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.786+0000 7f6e2af4d640 1 -- 172.21.15.193:0/2462815750 shutdown_connections 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.786+0000 7f6e2af4d640 1 -- 172.21.15.193:0/2462815750 wait complete. 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 10 monclient: build_initial_monmap 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 10 monclient: monmap: 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T19:40:43.791 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 1 librados: starting msgr at 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 1 librados: starting objecter 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 1 Processor -- start 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 1 -- start start 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 1 librados: setting wanted keys 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 1 librados: calling monclient init 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.787+0000 7f6e2af4d640 10 monclient: init 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e240a9920 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:43.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e240b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e24146450 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e24141340 con 0x7f6e240a9920 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e240ac550 con 0x7f6e240b45a0 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e24145120 con 0x7f6e24146450 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.788+0000 7f6e2af4d640 10 monclient(hunting): authenticate will time out at 4851.941261s 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.789+0000 7f6e23fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f6e240b45a0 0x7f6e241370e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52762/0 (socket says 172.21.15.193:52762) 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.789+0000 7f6e23fff640 1 -- 172.21.15.193:0/3810799420 learned_addr learned my addr 172.21.15.193:0/3810799420 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.789+0000 7f6e217fa640 10 client.?.objecter ms_handle_connect 0x7f6e240b45a0 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.789+0000 7f6e217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.789+0000 7f6e217fa640 10 client.?.objecter ms_handle_connect 0x7f6e240a9920 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.789+0000 7f6e217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.789+0000 7f6e217fa640 10 client.?.objecter ms_handle_connect 0x7f6e24146450 2024-03-25T19:40:43.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.789+0000 7f6e217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510239283 0 0) 0x7f6e24145120 con 0x7f6e24146450 2024-03-25T19:40:43.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): my global_id is 14150 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08004eb0 con 0x7f6e24146450 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247645412 0 0) 0x7f6e240ac550 con 0x7f6e240b45a0 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e24145120 con 0x7f6e240b45a0 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468260132 0 0) 0x7f6e24141340 con 0x7f6e240a9920 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:40:43.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.791+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e240ac550 con 0x7f6e240a9920 2024-03-25T19:40:43.796 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:43.792+0000 7f04fdc42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.795+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306068132 0 0) 0x7f6e24145120 con 0x7f6e240b45a0 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.795+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e24141340 con 0x7f6e240b45a0 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.795+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6e10001ae0 con 0x7f6e240b45a0 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.795+0000 7f6e217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.795+0000 7f6e217fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.795+0000 7f6e217fa640 10 monclient(hunting): dump: 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.797 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.795+0000 7f6e217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.796+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2570452956 0 0) 0x7f6e240ac550 con 0x7f6e240a9920 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.796+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e24145120 con 0x7f6e240a9920 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.796+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6e18001830 con 0x7f6e240a9920 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.796+0000 7f6e217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.796+0000 7f6e217fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.796+0000 7f6e217fa640 10 monclient(hunting): dump: 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.798 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.797+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 547198667 0 0) 0x7f6e08004eb0 con 0x7f6e24146450 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.797+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e240ac550 con 0x7f6e24146450 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.797+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6e1c001af0 con 0x7f6e24146450 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.797+0000 7f6e217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.797+0000 7f6e217fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.797+0000 7f6e217fa640 10 monclient(hunting): dump: 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3277638495 0 0) 0x7f6e24141340 con 0x7f6e240b45a0 2024-03-25T19:40:43.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 >> v1:172.21.15.71:6790/0 conn(0x7f6e24146450 legacy=0x7f6e241d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 >> v1:172.21.15.71:6789/0 conn(0x7f6e240a9920 legacy=0x7f6e241447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:43.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 1 monclient: found mon.b 2024-03-25T19:40:43.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24143bc0 con 0x7f6e240b45a0 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 10 monclient: _renew_subs 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e24143e10 con 0x7f6e240b45a0 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 10 monclient: _renew_subs 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e241379d0 con 0x7f6e240b45a0 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.799+0000 7f6e2af4d640 1 librados: init done 2024-03-25T19:40:43.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.800+0000 7f6e2af4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.800+0000 7f6e2af4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.800+0000 7f6e2af4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.800+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e24144f50 con 0x7f6e240b45a0 2024-03-25T19:40:43.899 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.899+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e10004a90 con 0x7f6e240b45a0 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.899+0000 7f6e217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.899+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6e10004c00 con 0x7f6e240b45a0 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.899+0000 7f6e217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.899+0000 7f6e217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.899+0000 7f6e217fa640 10 monclient: dump: 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.900 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.899+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6e100050f0 con 0x7f6e240b45a0 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.899+0000 7f6e22ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.900+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1590136594 0 0) 0x7f6e10006840 con 0x7f6e240b45a0 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.900+0000 7f6e217fa640 10 client.4156.objecter ms_dispatch 0x7f6e24000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.900+0000 7f6e217fa640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.900+0000 7f6e217fa640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.900+0000 7f6e217fa640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.900+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6e10006b80 con 0x7f6e240b45a0 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.900+0000 7f6e217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.901 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.900+0000 7f6e217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:43.973 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.973+0000 7f6e2af4d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d50af945-a618-4993-97bc-b141ea9f3237", "id": 6}] 2024-03-25T19:40:43.973 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.973+0000 7f6e2af4d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d50af945-a618-4993-97bc-b141ea9f3237", "id": 6}] 2024-03-25T19:40:43.973 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.973+0000 7f6e2af4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:43.973 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:43.973+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d50af945-a618-4993-97bc-b141ea9f3237", "id": 6} v 0) v1 -- 0x7f6e2404f730 con 0x7f6e240b45a0 2024-03-25T19:40:44.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.056+0000 7f6e217fa640 1 -- 172.21.15.193:0/3810799420 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d50af945-a618-4993-97bc-b141ea9f3237", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2397359023 0 1113811849) 0x7f6e10006320 con 0x7f6e240b45a0 2024-03-25T19:40:44.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.056+0000 7f6e217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d50af945-a618-4993-97bc-b141ea9f3237", "id": 6}] 2024-03-25T19:40:44.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.056+0000 7f6e217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:44.056 INFO:teuthology.orchestra.run.smithi193.stdout:6 2024-03-25T19:40:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 10 librados: watch_flush enter 2024-03-25T19:40:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 10 librados: watch_flush exit 2024-03-25T19:40:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T19:40:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 10 client.4156.objecter successfully canceled tick 2024-03-25T19:40:44.057 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 10 monclient: shutdown 2024-03-25T19:40:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 >> v1:172.21.15.193:6789/0 conn(0x7f6e240b45a0 legacy=0x7f6e241370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 shutdown_connections 2024-03-25T19:40:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 >> 172.21.15.193:0/3810799420 conn(0x7f6e240b7f40 msgr2=0x7f6e24138520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 shutdown_connections 2024-03-25T19:40:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 1 -- 172.21.15.193:0/3810799420 wait complete. 2024-03-25T19:40:44.058 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.057+0000 7f6e2af4d640 1 librados: shutdown 2024-03-25T19:40:44.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new f787a624-5765-4ba1-8afd-d5497f011748 7 2024-03-25T19:40:44.068 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:44.069+0000 7efe2da0e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:40:44.114 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:44.115+0000 7f04fdc42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:40:44.119 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:44.120+0000 7efe2da0e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:40:44.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.167+0000 7f259c274640 10 monclient: get_monmap_and_config 2024-03-25T19:40:44.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.167+0000 7f259c274640 10 monclient: build_initial_monmap 2024-03-25T19:40:44.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.167+0000 7f259c274640 10 monclient: monmap: 2024-03-25T19:40:44.167 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:44.167 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.167 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:40:44.168797+0000 2024-03-25T19:40:44.168 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:40:44.168797+0000 2024-03-25T19:40:44.168 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:44.168 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:44.168 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:44.168 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:44.168 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:44.168 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:44.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.168+0000 7f259c274640 1 Processor -- start 2024-03-25T19:40:44.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.168+0000 7f259c274640 1 -- start start 2024-03-25T19:40:44.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.168+0000 7f259c274640 10 monclient: init 2024-03-25T19:40:44.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 10 monclient(hunting): picked mon.noname-c con 0x7f2594070210 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 10 monclient(hunting): picked mon.noname-a con 0x7f25940b06d0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 10 monclient(hunting): picked mon.noname-b con 0x7f25940705f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25940b0b80 con 0x7f25940b06d0 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25940b0430 con 0x7f2594070210 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2594071670 con 0x7f25940705f0 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259c274640 10 monclient(hunting): authenticate will time out at 4852.321937s 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259a010640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f2594070210 0x7f2594140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52768/0 (socket says 172.21.15.193:52768) 2024-03-25T19:40:44.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.169+0000 7f259a010640 1 -- 172.21.15.193:0/2413793004 learned_addr learned my addr 172.21.15.193:0/2413793004 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:44.171 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:44.171+0000 7f04fdc42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929639432 0 0) 0x7f2594071670 con 0x7f25940705f0 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 10 monclient(hunting): _negotiate 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 10 monclient(hunting): my global_id is 14153 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2580003520 con 0x7f25940705f0 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461901970 0 0) 0x7f2590001110 con 0x7f2594070210 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 10 monclient(hunting): _negotiate 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:44.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:40:44.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.172+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2594071670 con 0x7f2594070210 2024-03-25T19:40:44.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.174+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256112729 0 0) 0x7f25940b0b80 con 0x7f25940b06d0 2024-03-25T19:40:44.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.174+0000 7f259900e640 10 monclient(hunting): _negotiate 2024-03-25T19:40:44.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.174+0000 7f259900e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:44.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.174+0000 7f259900e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:44.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.174+0000 7f259900e640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:40:44.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.174+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590001110 con 0x7f25940b06d0 2024-03-25T19:40:44.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165966943 0 0) 0x7f2594071670 con 0x7f2594070210 2024-03-25T19:40:44.175 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 >> v1:172.21.15.71:6790/0 conn(0x7f25940705f0 legacy=0x7f259414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 >> v1:172.21.15.71:6789/0 conn(0x7f25940b06d0 legacy=0x7f25940afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 10 monclient: _finish_hunting 0 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 1 monclient: found mon.noname-c 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2594070a60 con 0x7f2594070210 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 10 monclient: _finish_auth 0 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 10 monclient: _check_auth_tickets 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259900e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259c274640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.175+0000 7f259c274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:44.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.176+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2590001620 con 0x7f2594070210 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.176+0000 7f259900e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.176+0000 7f259900e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.176+0000 7f259900e640 10 monclient: dump: 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:44.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.176+0000 7f259c274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:44.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2590001800 con 0x7f2594070210 2024-03-25T19:40:44.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259900e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:44.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259900e640 1 -- 172.21.15.193:0/2413793004 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2590001b90 con 0x7f2594070210 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 4 set_mon_vals no callback set 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259900e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259900e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259900e640 10 monclient: dump: 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:44.181 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 10 monclient: shutdown 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 1 -- 172.21.15.193:0/2413793004 >> v1:172.21.15.193:6789/0 conn(0x7f2594070210 legacy=0x7f2594140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 1 -- 172.21.15.193:0/2413793004 shutdown_connections 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 1 -- 172.21.15.193:0/2413793004 >> 172.21.15.193:0/2413793004 conn(0x7f25940b5cd0 msgr2=0x7f25940b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 1 -- 172.21.15.193:0/2413793004 shutdown_connections 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.177+0000 7f259c274640 1 -- 172.21.15.193:0/2413793004 wait complete. 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.178+0000 7f259c274640 10 monclient: build_initial_monmap 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.178+0000 7f259c274640 10 monclient: monmap: 2024-03-25T19:40:44.182 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.178+0000 7f259c274640 1 librados: starting msgr at 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.178+0000 7f259c274640 1 librados: starting objecter 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.178+0000 7f259c274640 1 Processor -- start 2024-03-25T19:40:44.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 1 -- start start 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 1 librados: setting wanted keys 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 1 librados: calling monclient init 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 10 monclient: init 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 10 monclient(hunting): picked mon.noname-a con 0x7f2594070210 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 10 monclient(hunting): picked mon.noname-b con 0x7f25940705f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 10 monclient(hunting): picked mon.noname-c con 0x7f25940b06d0 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25941ccc50 con 0x7f2594070210 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259413aea0 con 0x7f25940705f0 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25941cc880 con 0x7f25940b06d0 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.179+0000 7f259c274640 10 monclient(hunting): authenticate will time out at 4852.332269s 2024-03-25T19:40:44.184 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.180+0000 7f259980f640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f25940705f0 0x7f2594133df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:52784/0 (socket says 172.21.15.193:52784) 2024-03-25T19:40:44.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.180+0000 7f259980f640 1 -- 172.21.15.193:0/403466187 learned_addr learned my addr 172.21.15.193:0/403466187 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T19:40:44.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.180+0000 7f258affd640 10 client.?.objecter ms_handle_connect 0x7f25940705f0 2024-03-25T19:40:44.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.180+0000 7f258affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:44.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.180+0000 7f258affd640 10 client.?.objecter ms_handle_connect 0x7f2594070210 2024-03-25T19:40:44.185 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.180+0000 7f258affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:44.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550159980 0 0) 0x7f259413aea0 con 0x7f25940705f0 2024-03-25T19:40:44.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:44.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2578004eb0 con 0x7f25940705f0 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133357142 0 0) 0x7f25941ccc50 con 0x7f2594070210 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.182+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259413aea0 con 0x7f2594070210 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.183+0000 7f258affd640 10 client.?.objecter ms_handle_connect 0x7f25940b06d0 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.183+0000 7f258affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.185+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230291325 0 0) 0x7f25941cc880 con 0x7f25940b06d0 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.185+0000 7f258affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:44.187 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.185+0000 7f258affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.185+0000 7f258affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.185+0000 7f258affd640 10 monclient(hunting): my global_id is 14156 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.185+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25941ccc50 con 0x7f25940b06d0 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.186+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2908310784 0 0) 0x7f2578004eb0 con 0x7f25940705f0 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.186+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25941cc880 con 0x7f25940705f0 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.186+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2584001970 con 0x7f25940705f0 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.186+0000 7f258affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.186+0000 7f258affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.186+0000 7f258affd640 10 monclient(hunting): dump: 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.188 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:44.189 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:44.189 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:44.189 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:44.189 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:44.189 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:44.189 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.186+0000 7f258affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:44.191 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4105254577 0 0) 0x7f259413aea0 con 0x7f2594070210 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2578004eb0 con 0x7f2594070210 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f25900047e0 con 0x7f2594070210 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 10 monclient(hunting): dump: 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.193 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1536772988 0 0) 0x7f25941cc880 con 0x7f25940705f0 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 >> v1:172.21.15.71:6790/0 conn(0x7f25940b06d0 legacy=0x7f25941d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 >> v1:172.21.15.71:6789/0 conn(0x7f2594070210 legacy=0x7f25941cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 1 monclient: found mon.b 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:44.194 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.189+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2594134500 con 0x7f25940705f0 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f258affd640 10 monclient: _finish_auth 0 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f258affd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f258affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 5 monclient: authenticate success, global_id 4162 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 10 monclient: _renew_subs 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2594134750 con 0x7f25940705f0 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 10 monclient: _renew_subs 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25940b48a0 con 0x7f25940705f0 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 1 librados: init done 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:44.195 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.190+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25941cca10 con 0x7f25940705f0 2024-03-25T19:40:44.239 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:44.240+0000 7efe2da0e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:40:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2584003610 con 0x7f25940705f0 2024-03-25T19:40:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2584003a20 con 0x7f25940705f0 2024-03-25T19:40:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 10 monclient: dump: 2024-03-25T19:40:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T19:40:44.287 INFO:teuthology.orchestra.run.smithi193.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f259880d640 4 set_mon_vals no callback set 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2584003fd0 con 0x7f25940705f0 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3809455686 0 0) 0x7f2584006900 con 0x7f25940705f0 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 10 client.4162.objecter ms_dispatch 0x7f2594000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:40:44.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:40:44.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:40:44.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T19:40:44.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2584018de0 con 0x7f25940705f0 2024-03-25T19:40:44.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:44.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.287+0000 7f258affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:44.291 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T19:40:44.292+0000 7f04fdc42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:40:44.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.359+0000 7f259c274640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f787a624-5765-4ba1-8afd-d5497f011748", "id": 7}] 2024-03-25T19:40:44.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.359+0000 7f259c274640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f787a624-5765-4ba1-8afd-d5497f011748", "id": 7}] 2024-03-25T19:40:44.359 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.359+0000 7f259c274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:44.359 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.359+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f787a624-5765-4ba1-8afd-d5497f011748", "id": 7} v 0) v1 -- 0x7f259404f730 con 0x7f25940705f0 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.527+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2363419029 0 0) 0x7f2584004280 con 0x7f25940705f0 2024-03-25T19:40:44.744 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.744+0000 7f258affd640 1 -- 172.21.15.193:0/403466187 <== mon.1 v1:172.21.15.193:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f787a624-5765-4ba1-8afd-d5497f011748", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2500611453 0 1371665406) 0x7f2584016320 con 0x7f25940705f0 2024-03-25T19:40:44.744 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f258affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f787a624-5765-4ba1-8afd-d5497f011748", "id": 7}] 2024-03-25T19:40:44.744 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f258affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:44.745 INFO:teuthology.orchestra.run.smithi193.stdout:7 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f259c274640 10 librados: watch_flush enter 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f259c274640 10 librados: watch_flush exit 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f259c274640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f259c274640 10 client.4162.objecter successfully canceled tick 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f259c274640 10 monclient: shutdown 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f259c274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.745+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 >> v1:172.21.15.193:6789/0 conn(0x7f25940705f0 legacy=0x7f2594133df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.746+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 shutdown_connections 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.746+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 >> 172.21.15.193:0/403466187 conn(0x7f25940b5cd0 msgr2=0x7f25940ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.746+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 shutdown_connections 2024-03-25T19:40:44.746 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.746+0000 7f259c274640 1 -- 172.21.15.193:0/403466187 wait complete. 2024-03-25T19:40:44.747 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T19:40:44.746+0000 7f259c274640 1 librados: shutdown 2024-03-25T19:40:44.755 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:44.755 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:40:44.756 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:40:44.758 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:40:44.758 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:44.758 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:40:44.758 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:40:44.760 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:40:44.760 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:44.760 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:40:44.760 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:40:44.762 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:40:44.762 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:44.763 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:40:44.763 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:40:44.765 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:40:44.765 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:44.765 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:40:44.765 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:40:44.767 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:40:44.767 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:44.768 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:40:44.768 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:40:44.770 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:40:44.770 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:44.770 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:40:44.770 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:40:44.772 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:40:44.772 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:44.772 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:40:44.772 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:40:44.774 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:40:44.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.936+0000 7f8ad2502640 10 monclient: get_monmap_and_config 2024-03-25T19:40:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.936+0000 7f8ad2502640 10 monclient: build_initial_monmap 2024-03-25T19:40:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.936+0000 7f8ad2502640 10 monclient: monmap: 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:40:44.937801+0000 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:40:44.937801+0000 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:44.937 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.937+0000 7f8ad2502640 1 Processor -- start 2024-03-25T19:40:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.938+0000 7f8ad2502640 1 -- start start 2024-03-25T19:40:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.938+0000 7f8ad2502640 10 monclient: init 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.938+0000 7f8ad2502640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.938+0000 7f8ad2502640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.939+0000 7f8ad2502640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac40389d0 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.939+0000 7f8ad2502640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac4038db0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.939+0000 7f8ad2502640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac4030650 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.939+0000 7f8ad2502640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac40c54b0 con 0x7f8ac40389d0 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.939+0000 7f8ad2502640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac4036420 con 0x7f8ac4038db0 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.939+0000 7f8ad2502640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac4039830 con 0x7f8ac4030650 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.939+0000 7f8ad2502640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.939+0000 7f8ad2502640 10 monclient(hunting): authenticate will time out at 4851.250935s 2024-03-25T19:40:44.940 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.940+0000 7f8ad0cff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f8ac4038db0 0x7f8ac40cadc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:49488/0 (socket says 172.21.15.71:49488) 2024-03-25T19:40:44.940 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.940+0000 7f8ad0cff640 1 -- 172.21.15.71:0/3651152445 learned_addr learned my addr 172.21.15.71:0/3651152445 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:40:44.948 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.949+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375499849 0 0) 0x7f8ac4036420 con 0x7f8ac4038db0 2024-03-25T19:40:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.949+0000 7f8ac3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.949+0000 7f8ac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.949+0000 7f8ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.949+0000 7f8ac3fff640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:40:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.949+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab8003520 con 0x7f8ac4038db0 2024-03-25T19:40:44.950 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.951+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090773724 0 0) 0x7f8ac4039830 con 0x7f8ac4030650 2024-03-25T19:40:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.951+0000 7f8ac3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.951+0000 7f8ac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.951+0000 7f8ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.951+0000 7f8ac3fff640 10 monclient(hunting): my global_id is 14177 2024-03-25T19:40:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.951+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac4036420 con 0x7f8ac4030650 2024-03-25T19:40:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165876441 0 0) 0x7f8ab8003520 con 0x7f8ac4038db0 2024-03-25T19:40:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 >> v1:172.21.15.71:6790/0 conn(0x7f8ac4030650 legacy=0x7f8ac40d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 >> v1:172.21.15.71:6789/0 conn(0x7f8ac40389d0 legacy=0x7f8ac4033510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 1 monclient: found mon.noname-c 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac4035680 con 0x7f8ac4038db0 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8ac8001690 con 0x7f8ac4038db0 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ac3fff640 10 monclient: dump: 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-25T19:40:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ad2502640 5 monclient: authenticate success, global_id 4177 2024-03-25T19:40:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.958+0000 7f8ad2502640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:44.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.981+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ac80017f0 con 0x7f8ac4038db0 2024-03-25T19:40:44.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.981+0000 7f8ac3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:44.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.981+0000 7f8ac3fff640 1 -- 172.21.15.71:0/3651152445 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8ac8001c00 con 0x7f8ac4038db0 2024-03-25T19:40:44.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.981+0000 7f8ad2502640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:44.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.981+0000 7f8ad2502640 4 set_mon_vals no callback set 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.981+0000 7f8ac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.981+0000 7f8ac3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.981+0000 7f8ac3fff640 10 monclient: dump: 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:44.981 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.982+0000 7f8ad2502640 10 monclient: shutdown 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.982+0000 7f8ad2502640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.982+0000 7f8ad2502640 1 -- 172.21.15.71:0/3651152445 >> v1:172.21.15.193:6789/0 conn(0x7f8ac4038db0 legacy=0x7f8ac40cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.982+0000 7f8ad2502640 1 -- 172.21.15.71:0/3651152445 shutdown_connections 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.982+0000 7f8ad2502640 1 -- 172.21.15.71:0/3651152445 >> 172.21.15.71:0/3651152445 conn(0x7f8ac4005bf0 msgr2=0x7f8ac4006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.982+0000 7f8ad2502640 1 -- 172.21.15.71:0/3651152445 shutdown_connections 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.982+0000 7f8ad2502640 1 -- 172.21.15.71:0/3651152445 wait complete. 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 10 monclient: build_initial_monmap 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 10 monclient: monmap: 2024-03-25T19:40:44.983 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 1 librados: starting msgr at 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 1 librados: starting objecter 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 1 Processor -- start 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 1 -- start start 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 1 librados: setting wanted keys 2024-03-25T19:40:44.984 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 1 librados: calling monclient init 2024-03-25T19:40:44.985 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.983+0000 7f8ad2502640 10 monclient: init 2024-03-25T19:40:44.985 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.984+0000 7f8ad2502640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.984+0000 7f8ad2502640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.984+0000 7f8ad2502640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac4030650 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad2502640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac40389d0 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad2502640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac402eff0 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad2502640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac402ebd0 con 0x7f8ac4030650 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad2502640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac415fd00 con 0x7f8ac402eff0 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad2502640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac4160930 con 0x7f8ac40389d0 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad2502640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad2502640 10 monclient(hunting): authenticate will time out at 4851.297074s 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad0cff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8ac40389d0 0x7f8ac4160220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52952/0 (socket says 172.21.15.71:52952) 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.985+0000 7f8ad0cff640 1 -- 172.21.15.71:0/3216336038 learned_addr learned my addr 172.21.15.71:0/3216336038 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:40:44.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.986+0000 7f8ac1ffb640 10 client.?.objecter ms_handle_connect 0x7f8ac40389d0 2024-03-25T19:40:44.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.986+0000 7f8ac1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:44.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.986+0000 7f8ac1ffb640 10 client.?.objecter ms_handle_connect 0x7f8ac402eff0 2024-03-25T19:40:44.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:44.986+0000 7f8ac1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:45.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.033+0000 7f8ac1ffb640 10 client.?.objecter ms_handle_connect 0x7f8ac4030650 2024-03-25T19:40:45.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.033+0000 7f8ac1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:45.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.142+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050611186 0 0) 0x7f8ac415fd00 con 0x7f8ac402eff0 2024-03-25T19:40:45.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.142+0000 7f8ac1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:45.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.142+0000 7f8ac1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:45.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.142+0000 7f8ac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:45.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.142+0000 7f8ac1ffb640 10 monclient(hunting): my global_id is 4180 2024-03-25T19:40:45.142 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.142+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab0004eb0 con 0x7f8ac402eff0 2024-03-25T19:40:45.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.150+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3906448558 0 0) 0x7f8ab0004eb0 con 0x7f8ac402eff0 2024-03-25T19:40:45.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.150+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ac415fd00 con 0x7f8ac402eff0 2024-03-25T19:40:45.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.150+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8abc001930 con 0x7f8ac402eff0 2024-03-25T19:40:45.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.150+0000 7f8ac1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:45.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.150+0000 7f8ac1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:45.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.150+0000 7f8ac1ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:45.150 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.150+0000 7f8ac1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.151+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934472417 0 0) 0x7f8ac4160930 con 0x7f8ac40389d0 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.151+0000 7f8ac1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.151+0000 7f8ac1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.151+0000 7f8ac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.151+0000 7f8ac1ffb640 10 monclient(hunting): my global_id is 14180 2024-03-25T19:40:45.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.151+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab0004eb0 con 0x7f8ac40389d0 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3616534299 0 0) 0x7f8ac415fd00 con 0x7f8ac402eff0 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 >> v1:172.21.15.71:6790/0 conn(0x7f8ac40389d0 legacy=0x7f8ac4160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 >> v1:172.21.15.71:6789/0 conn(0x7f8ac4030650 legacy=0x7f8ac402d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 1 monclient: found mon.b 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac4160ac0 con 0x7f8ac402eff0 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ac1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 5 monclient: authenticate success, global_id 4180 2024-03-25T19:40:45.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 10 monclient: _renew_subs 2024-03-25T19:40:45.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:45.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac40c83a0 con 0x7f8ac402eff0 2024-03-25T19:40:45.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:45.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 10 monclient: _renew_subs 2024-03-25T19:40:45.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:45.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac402fdb0 con 0x7f8ac402eff0 2024-03-25T19:40:45.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.157+0000 7f8ad2502640 1 librados: init done 2024-03-25T19:40:45.159 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.159+0000 7f8ad2502640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:45.159 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.159+0000 7f8ad2502640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:45.159 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.159+0000 7f8ad2502640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:45.159 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.159+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac4007b80 con 0x7f8ac402eff0 2024-03-25T19:40:45.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.345+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8abc003610 con 0x7f8ac402eff0 2024-03-25T19:40:45.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.345+0000 7f8ac1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:45.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.345+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8abc003920 con 0x7f8ac402eff0 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.345+0000 7f8ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.345+0000 7f8ac1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.345+0000 7f8ac1ffb640 10 monclient: dump: 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:45.346 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:40:45.347 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.346+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2363419029 0 0) 0x7f8abc00ea00 con 0x7f8ac402eff0 2024-03-25T19:40:45.347 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.345+0000 7f8ac37fe640 4 set_mon_vals no callback set 2024-03-25T19:40:45.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.349+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f8abc014470 con 0x7f8ac402eff0 2024-03-25T19:40:45.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.349+0000 7f8ac1ffb640 10 client.4180.objecter ms_dispatch 0x7f8acc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:45.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.349+0000 7f8ac1ffb640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:45.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.349+0000 7f8ac1ffb640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:45.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.350+0000 7f8ac1ffb640 20 client.4180.objecter dump_active .. 0 homeless 2024-03-25T19:40:45.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.350+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8abc013730 con 0x7f8ac402eff0 2024-03-25T19:40:45.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.350+0000 7f8ac1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:45.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.350+0000 7f8ac1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:45.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.469+0000 7f8ad2502640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:45.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.469+0000 7f8ad2502640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:45.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.469+0000 7f8ad2502640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:45.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.469+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ac415fe90 con 0x7f8ac402eff0 2024-03-25T19:40:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.556+0000 7f8ac1ffb640 1 -- 172.21.15.71:0/3216336038 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f8abc011f10 con 0x7f8ac402eff0 2024-03-25T19:40:45.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.556+0000 7f8ac1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:45.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.556+0000 7f8ac1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:45.556 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-25T19:40:45.556 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 10 librados: watch_flush enter 2024-03-25T19:40:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 10 librados: watch_flush exit 2024-03-25T19:40:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 20 client.4180.objecter shutdown clearing up homeless session... 2024-03-25T19:40:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 10 client.4180.objecter successfully canceled tick 2024-03-25T19:40:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 10 monclient: shutdown 2024-03-25T19:40:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 >> v1:172.21.15.193:6789/0 conn(0x7f8ac402eff0 legacy=0x7f8ac402f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 shutdown_connections 2024-03-25T19:40:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 >> 172.21.15.71:0/3216336038 conn(0x7f8ac4005bf0 msgr2=0x7f8ac4034080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 shutdown_connections 2024-03-25T19:40:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.557+0000 7f8ad2502640 1 -- 172.21.15.71:0/3216336038 wait complete. 2024-03-25T19:40:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-25T19:40:45.558+0000 7f8ad2502640 1 librados: shutdown 2024-03-25T19:40:45.571 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:40:45.572 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:40:46.061 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:46.061+0000 7efdce2c2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:40:46.146 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:40:46.147+0000 7efdce2c2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:40:46.377 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.521+0000 7f572650d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.521+0000 7f572650d640 10 monclient: build_initial_monmap 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.521+0000 7f572650d640 10 monclient: monmap: 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:40:46.522472+0000 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:40:46.522472+0000 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:46.521 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:46.522 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:46.522 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:46.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.522+0000 7f572650d640 1 Processor -- start 2024-03-25T19:40:46.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.522+0000 7f572650d640 1 -- start start 2024-03-25T19:40:46.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.522+0000 7f572650d640 10 monclient: init 2024-03-25T19:40:46.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.523+0000 7f572650d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:46.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.523+0000 7f572650d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:46.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.523+0000 7f572650d640 10 monclient(hunting): picked mon.noname-c con 0x7f57200b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.523+0000 7f572650d640 10 monclient(hunting): picked mon.noname-a con 0x7f57200b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.524+0000 7f572650d640 10 monclient(hunting): picked mon.noname-b con 0x7f572014be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.524+0000 7f572650d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200af940 con 0x7f57200b1730 2024-03-25T19:40:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.524+0000 7f572650d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200b0a70 con 0x7f57200b4d60 2024-03-25T19:40:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.524+0000 7f572650d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200b0d50 con 0x7f572014be90 2024-03-25T19:40:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.524+0000 7f572650d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.524+0000 7f572650d640 10 monclient(hunting): authenticate will time out at 4852.835927s 2024-03-25T19:40:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.524+0000 7f571f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f57200b1730 0x7f57200b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52170/0 (socket says 172.21.15.71:52170) 2024-03-25T19:40:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.524+0000 7f571f7fe640 1 -- 172.21.15.71:0/1557901650 learned_addr learned my addr 172.21.15.71:0/1557901650 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340418132 0 0) 0x7f57200af940 con 0x7f57200b1730 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): my global_id is 4173 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5708003520 con 0x7f57200b1730 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232669861 0 0) 0x7f57200b0a70 con 0x7f57200b4d60 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): my global_id is 4183 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57200af940 con 0x7f57200b4d60 2024-03-25T19:40:46.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112578042 0 0) 0x7f57200b0d50 con 0x7f572014be90 2024-03-25T19:40:46.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 10 monclient(hunting): my global_id is 14183 2024-03-25T19:40:46.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.528+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57200b0a70 con 0x7f572014be90 2024-03-25T19:40:46.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4042117986 0 0) 0x7f57200af940 con 0x7f57200b4d60 2024-03-25T19:40:46.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 >> v1:172.21.15.71:6790/0 conn(0x7f572014be90 legacy=0x7f572014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 >> v1:172.21.15.71:6789/0 conn(0x7f57200b1730 legacy=0x7f57200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 1 monclient: found mon.noname-c 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57200b1030 con 0x7f57200b4d60 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 10 monclient: _finish_auth 0 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f571effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f572650d640 5 monclient: authenticate success, global_id 4183 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.532+0000 7f572650d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.533+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f57100015b0 con 0x7f57200b4d60 2024-03-25T19:40:46.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.533+0000 7f571effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.533+0000 7f571effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.533+0000 7f571effd640 10 monclient: dump: 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:46.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.533+0000 7f572650d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.535+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5710001800 con 0x7f57200b4d60 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.535+0000 7f571effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.535+0000 7f572650d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.535+0000 7f572650d640 4 set_mon_vals no callback set 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f571effd640 1 -- 172.21.15.71:0/1557901650 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5710001b90 con 0x7f57200b4d60 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f571effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f571effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f572650d640 10 monclient: shutdown 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f571effd640 10 monclient: dump: 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:46.538 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f572650d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f572650d640 1 -- 172.21.15.71:0/1557901650 >> v1:172.21.15.193:6789/0 conn(0x7f57200b4d60 legacy=0x7f572013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f572650d640 1 -- 172.21.15.71:0/1557901650 shutdown_connections 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f572650d640 1 -- 172.21.15.71:0/1557901650 >> 172.21.15.71:0/1557901650 conn(0x7f57200617a0 msgr2=0x7f5720061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f572650d640 1 -- 172.21.15.71:0/1557901650 shutdown_connections 2024-03-25T19:40:46.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.536+0000 7f572650d640 1 -- 172.21.15.71:0/1557901650 wait complete. 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.537+0000 7f572650d640 10 monclient: build_initial_monmap 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.537+0000 7f572650d640 10 monclient: monmap: 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.538+0000 7f572650d640 1 librados: starting msgr at 2024-03-25T19:40:46.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.538+0000 7f572650d640 1 librados: starting objecter 2024-03-25T19:40:46.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.538+0000 7f572650d640 1 Processor -- start 2024-03-25T19:40:46.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.538+0000 7f572650d640 1 -- start start 2024-03-25T19:40:46.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.538+0000 7f572650d640 1 librados: setting wanted keys 2024-03-25T19:40:46.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.538+0000 7f572650d640 1 librados: calling monclient init 2024-03-25T19:40:46.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.538+0000 7f572650d640 10 monclient: init 2024-03-25T19:40:46.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:46.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:46.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 10 monclient(hunting): picked mon.noname-b con 0x7f57200b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:46.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 10 monclient(hunting): picked mon.noname-c con 0x7f57200b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:46.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 10 monclient(hunting): picked mon.noname-a con 0x7f572014be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572013a530 con 0x7f572014be90 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572013a810 con 0x7f57200b1730 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572013aaf0 con 0x7f57200b4d60 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.539+0000 7f572650d640 10 monclient(hunting): authenticate will time out at 4852.851635s 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.540+0000 7f571ffff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f57200b1730 0x7f5720140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:38364/0 (socket says 172.21.15.71:38364) 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.540+0000 7f571ffff640 1 -- 172.21.15.71:0/4143801468 learned_addr learned my addr 172.21.15.71:0/4143801468 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.540+0000 7f571cff9640 10 client.?.objecter ms_handle_connect 0x7f57200b1730 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.540+0000 7f571cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.540+0000 7f571cff9640 10 client.?.objecter ms_handle_connect 0x7f57200b4d60 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.540+0000 7f571cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.540+0000 7f571cff9640 10 client.?.objecter ms_handle_connect 0x7f572014be90 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.540+0000 7f571cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.543+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293313040 0 0) 0x7f572013a810 con 0x7f57200b1730 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.543+0000 7f571cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.543+0000 7f571cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.543+0000 7f571cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.543+0000 7f571cff9640 10 monclient(hunting): my global_id is 4186 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.543+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c004eb0 con 0x7f57200b1730 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150027944 0 0) 0x7f572013a530 con 0x7f572014be90 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 10 monclient(hunting): my global_id is 4176 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572013a810 con 0x7f572014be90 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632058279 0 0) 0x7f572013aaf0 con 0x7f57200b4d60 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 10 monclient(hunting): my global_id is 14186 2024-03-25T19:40:46.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.544+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572013a530 con 0x7f57200b4d60 2024-03-25T19:40:46.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.550+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2057162372 0 0) 0x7f570c004eb0 con 0x7f57200b1730 2024-03-25T19:40:46.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.551+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f572013aaf0 con 0x7f57200b1730 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.551+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4153669187 0 0) 0x7f572013a530 con 0x7f57200b4d60 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.551+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f570c004eb0 con 0x7f57200b4d60 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.551+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2535654303 0 0) 0x7f572013a810 con 0x7f572014be90 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.551+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f572013a530 con 0x7f572014be90 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5718001a80 con 0x7f572014be90 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 10 monclient(hunting): dump: 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.552 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5714001910 con 0x7f57200b4d60 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.552+0000 7f571cff9640 10 monclient(hunting): dump: 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.553 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.554 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:46.554 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:46.554 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:46.554 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:46.554 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:46.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.555+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f57100035f0 con 0x7f57200b1730 2024-03-25T19:40:46.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.555+0000 7f571cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.555+0000 7f571cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.555+0000 7f571cff9640 10 monclient(hunting): dump: 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.555+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1083327780 0 0) 0x7f572013aaf0 con 0x7f57200b1730 2024-03-25T19:40:46.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.555+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 >> v1:172.21.15.71:6790/0 conn(0x7f57200b4d60 legacy=0x7f572013af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.555+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 >> v1:172.21.15.71:6789/0 conn(0x7f572014be90 legacy=0x7f572013b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f571cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f571cff9640 1 monclient: found mon.b 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f571cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f571cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5720142b30 con 0x7f57200b1730 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f571cff9640 10 monclient: _finish_auth 0 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f571cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f571cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f572650d640 5 monclient: authenticate success, global_id 4186 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f572650d640 10 monclient: _renew_subs 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f572650d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572013bd30 con 0x7f57200b1730 2024-03-25T19:40:46.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f572650d640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:46.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f572650d640 10 monclient: _renew_subs 2024-03-25T19:40:46.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f572650d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:46.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.556+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5720139160 con 0x7f57200b1730 2024-03-25T19:40:46.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.557+0000 7f572650d640 1 librados: init done 2024-03-25T19:40:46.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.558+0000 7f572650d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.558+0000 7f572650d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.558+0000 7f572650d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:46.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:46.558+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57201394b0 con 0x7f57200b1730 2024-03-25T19:40:47.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.006+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5710001180 con 0x7f57200b1730 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.006+0000 7f571cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.006+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5710003c60 con 0x7f57200b1730 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.006+0000 7f571cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.006+0000 7f571cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.006+0000 7f571cff9640 10 monclient: dump: 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.007 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:47.008 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:47.008 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:47.008 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:47.008 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:47.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.006+0000 7f571e7fc640 4 set_mon_vals no callback set 2024-03-25T19:40:47.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.007+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 4208398109 0 0) 0x7f5710003e80 con 0x7f57200b1730 2024-03-25T19:40:47.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.008+0000 7f571f7fe640 10 monclient: get_auth_request con 0x7f570c06e050 auth_method 0 2024-03-25T19:40:47.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.008+0000 7f571cff9640 10 client.4186.objecter ms_handle_connect 0x7f570c06e050 2024-03-25T19:40:47.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.016+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f57100548e0 con 0x7f57200b1730 2024-03-25T19:40:47.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.016+0000 7f571cff9640 10 client.4186.objecter ms_dispatch 0x7f5720000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:47.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.016+0000 7f571cff9640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:47.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.016+0000 7f571cff9640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:47.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.016+0000 7f571cff9640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-25T19:40:47.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.017+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57100503f0 con 0x7f57200b1730 2024-03-25T19:40:47.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.017+0000 7f571cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.017+0000 7f571cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:47.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.384+0000 7f572650d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:47.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.384+0000 7f572650d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:47.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.384+0000 7f572650d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:47.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.384+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57200627b0 con 0x7f57200b1730 2024-03-25T19:40:47.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.716+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f571004fea0 con 0x7f57200b1730 2024-03-25T19:40:47.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.720+0000 7f571cff9640 1 -- 172.21.15.71:0/4143801468 <== mon.1 v1:172.21.15.193:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f5710003ac0 con 0x7f57200b1730 2024-03-25T19:40:47.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.720+0000 7f571cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:47.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.720+0000 7f571cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:47.720 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:40:47.720 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 10 librados: watch_flush enter 2024-03-25T19:40:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 10 librados: watch_flush exit 2024-03-25T19:40:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-25T19:40:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 10 client.4186.objecter successfully canceled tick 2024-03-25T19:40:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 >> v1:172.21.15.71:6800/3911735624 conn(0x7f570c06e050 legacy=0x7f570c070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 10 monclient: shutdown 2024-03-25T19:40:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 >> v1:172.21.15.193:6789/0 conn(0x7f57200b1730 legacy=0x7f5720140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 shutdown_connections 2024-03-25T19:40:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.723+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 >> 172.21.15.71:0/4143801468 conn(0x7f57200617a0 msgr2=0x7f57200b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.724+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 shutdown_connections 2024-03-25T19:40:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.724+0000 7f572650d640 1 -- 172.21.15.71:0/4143801468 wait complete. 2024-03-25T19:40:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:47.724+0000 7f572650d640 1 librados: shutdown 2024-03-25T19:40:47.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:51.718 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:06.869 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.718 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:06.880 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.733 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:06.897 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.790 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:06.941 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.790 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:06.952 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.796 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:06.960 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.849 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:06.999 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.880 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:07.033 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.912 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:07.062 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:51.942 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:07.095 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.003 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:07.149 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.052 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:07.217 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.058 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:07.219 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.063 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:07.209 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.113 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:07.278 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.116 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:07.277 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.327 INFO:tasks.ceph.osd.5.smithi193.stderr:2024-03-25T19:40:52.321+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:52.331 INFO:tasks.ceph.osd.6.smithi193.stderr:2024-03-25T19:40:52.326+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:52.374 INFO:tasks.ceph.osd.4.smithi193.stderr:2024-03-25T19:40:52.367+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:52.461 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-03-25T19:40:52.456+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:52.493 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-03-25T19:40:52.488+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:52.633 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-03-25T19:40:52.627+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:52.695 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-03-25T19:40:52.689+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:52.708 INFO:tasks.ceph.osd.7.smithi193.stderr:2024-03-25T19:40:52.702+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:40:52.785 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:07.946 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.820 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:07.971 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.835 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:07.999 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.930 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:08.080 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:52.963 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:08.116 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.115 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:08.261 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.164 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:08.326 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.165 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:08.327 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.173 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:08.338 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.192 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:08.343 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.241 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:08.405 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.294 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:08.444 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.339 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:08.493 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.484 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:08.630 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.523 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:08.684 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.532 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:08.694 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.536 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:08.701 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.553 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:08.705 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.607 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:08.770 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.610 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:08.760 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.652 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:08.805 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.788 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:08.949 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.796 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:08.942 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.843 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:08.995 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.851 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:09.016 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.874 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:09.025 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.892 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:09.054 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.900 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:09.064 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:53.917 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:09.070 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.052 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:09.213 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.064 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:09.210 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.115 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:09.280 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.132 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:09.284 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.142 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:09.292 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.181 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:09.334 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.186 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:09.348 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.192 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:09.356 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.319 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:09.480 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.332 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:09.478 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.386 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:09.551 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.411 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:09.561 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.420 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:09.572 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.445 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:09.598 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.485 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:09.647 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.488 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:09.652 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.544 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:54.584 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:09.745 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.610 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:09.756 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.634+0000 7faff5ad9640 10 monclient: get_monmap_and_config 2024-03-25T19:40:54.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.634+0000 7faff5ad9640 10 monclient: build_initial_monmap 2024-03-25T19:40:54.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.635+0000 7faff5ad9640 10 monclient: monmap: 2024-03-25T19:40:54.634 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:40:54.635890+0000 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:40:54.635890+0000 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:40:54.635 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:54.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.635+0000 7faff5ad9640 1 Processor -- start 2024-03-25T19:40:54.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.635+0000 7faff5ad9640 1 -- start start 2024-03-25T19:40:54.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.635+0000 7faff5ad9640 10 monclient: init 2024-03-25T19:40:54.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 10 monclient(hunting): picked mon.noname-a con 0x7faff006ea40 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 10 monclient(hunting): picked mon.noname-c con 0x7faff0071800 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 10 monclient(hunting): picked mon.noname-b con 0x7faff0071c40 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff006f1f0 con 0x7faff006ea40 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff0070d00 con 0x7faff0071800 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff00720b0 con 0x7faff0071c40 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7faff5ad9640 10 monclient(hunting): authenticate will time out at 4860.948260s 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.636+0000 7fafeffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7faff0071c40 0x7faff014a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:58154/0 (socket says 172.21.15.71:58154) 2024-03-25T19:40:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.637+0000 7fafeffff640 1 -- 172.21.15.71:0/3856604349 learned_addr learned my addr 172.21.15.71:0/3856604349 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:40:54.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285213425 0 0) 0x7faff00720b0 con 0x7faff0071c40 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): my global_id is 14213 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd8003520 con 0x7faff0071c40 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320725415 0 0) 0x7faff0070d00 con 0x7faff0071800 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff00720b0 con 0x7faff0071800 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458018216 0 0) 0x7faff006f1f0 con 0x7faff006ea40 2024-03-25T19:40:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 10 monclient(hunting): my global_id is 4203 2024-03-25T19:40:54.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.639+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff0070d00 con 0x7faff006ea40 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91564529 0 0) 0x7faff00720b0 con 0x7faff0071800 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 >> v1:172.21.15.71:6790/0 conn(0x7faff0071c40 legacy=0x7faff014a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 >> v1:172.21.15.71:6789/0 conn(0x7faff006ea40 legacy=0x7faff013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 1 monclient: found mon.noname-c 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff0072780 con 0x7faff0071800 2024-03-25T19:40:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fafe4001670 con 0x7faff0071800 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7faff5ad9640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7faff5ad9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7fafee7fc640 10 monclient: dump: 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:54.643 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:54.644 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:54.644 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:54.644 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:54.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.642+0000 7faff5ad9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:54.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafe40017d0 con 0x7faff0071800 2024-03-25T19:40:54.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7fafee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7fafee7fc640 1 -- 172.21.15.71:0/3856604349 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fafe4001be0 con 0x7faff0071800 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7fafee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7fafee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7fafee7fc640 10 monclient: dump: 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:54.647 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 4 set_mon_vals no callback set 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 10 monclient: shutdown 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 1 -- 172.21.15.71:0/3856604349 >> v1:172.21.15.193:6789/0 conn(0x7faff0071800 legacy=0x7faff0147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 1 -- 172.21.15.71:0/3856604349 shutdown_connections 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 1 -- 172.21.15.71:0/3856604349 >> 172.21.15.71:0/3856604349 conn(0x7faff006b630 msgr2=0x7faff00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 1 -- 172.21.15.71:0/3856604349 shutdown_connections 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.644+0000 7faff5ad9640 1 -- 172.21.15.71:0/3856604349 wait complete. 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.645+0000 7faff5ad9640 10 monclient: build_initial_monmap 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.645+0000 7faff5ad9640 10 monclient: monmap: 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:40:54.648 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.645+0000 7faff5ad9640 1 librados: starting msgr at 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.645+0000 7faff5ad9640 1 librados: starting objecter 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.645+0000 7faff5ad9640 1 Processor -- start 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 1 -- start start 2024-03-25T19:40:54.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 1 librados: setting wanted keys 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 1 librados: calling monclient init 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 10 monclient: init 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 10 monclient(hunting): picked mon.noname-b con 0x7faff006ea40 addr v1:172.21.15.193:6789/0 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 10 monclient(hunting): picked mon.noname-c con 0x7faff0071800 addr v1:172.21.15.71:6790/0 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 10 monclient(hunting): picked mon.noname-a con 0x7faff0071c40 addr v1:172.21.15.71:6789/0 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff01d3800 con 0x7faff0071c40 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff00ad720 con 0x7faff006ea40 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff013bb80 con 0x7faff0071800 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.646+0000 7faff5ad9640 10 monclient(hunting): authenticate will time out at 4860.958597s 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.647+0000 7fafeeffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7faff0071800 0x7faff013ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:58156/0 (socket says 172.21.15.71:58156) 2024-03-25T19:40:54.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.647+0000 7fafeeffd640 1 -- 172.21.15.71:0/1417486951 learned_addr learned my addr 172.21.15.71:0/1417486951 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:40:54.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.647+0000 7fafd7fff640 10 client.?.objecter ms_handle_connect 0x7faff006ea40 2024-03-25T19:40:54.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.647+0000 7fafd7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.647+0000 7fafd7fff640 10 client.?.objecter ms_handle_connect 0x7faff0071800 2024-03-25T19:40:54.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.647+0000 7fafd7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.647+0000 7fafd7fff640 10 client.?.objecter ms_handle_connect 0x7faff0071c40 2024-03-25T19:40:54.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.647+0000 7fafd7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775640869 0 0) 0x7faff013bb80 con 0x7faff0071800 2024-03-25T19:40:54.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): my global_id is 14216 2024-03-25T19:40:54.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafdc004eb0 con 0x7faff0071800 2024-03-25T19:40:54.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570942912 0 0) 0x7faff00ad720 con 0x7faff006ea40 2024-03-25T19:40:54.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff013bb80 con 0x7faff006ea40 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871791981 0 0) 0x7faff01d3800 con 0x7faff0071c40 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:40:54.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.649+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff00ad720 con 0x7faff0071c40 2024-03-25T19:40:54.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.653+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2719575260 0 0) 0x7faff013bb80 con 0x7faff006ea40 2024-03-25T19:40:54.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.653+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faff01d3800 con 0x7faff006ea40 2024-03-25T19:40:54.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.653+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fafe0001820 con 0x7faff006ea40 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.653+0000 7fafd7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.653+0000 7fafd7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.653+0000 7fafd7fff640 10 monclient(hunting): dump: 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:54.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.653+0000 7fafd7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2833861853 0 0) 0x7fafdc004eb0 con 0x7faff0071800 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faff013bb80 con 0x7faff0071800 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fafe4001670 con 0x7faff0071800 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 10 monclient(hunting): dump: 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:54.656 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034472251 0 0) 0x7faff00ad720 con 0x7faff0071c40 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafdc004eb0 con 0x7faff0071c40 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fafe8001a30 con 0x7faff0071c40 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.654+0000 7fafd7fff640 10 monclient(hunting): dump: 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:54.657 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:54.658 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:54.658 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:54.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780249329 0 0) 0x7faff01d3800 con 0x7faff006ea40 2024-03-25T19:40:54.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 >> v1:172.21.15.71:6790/0 conn(0x7faff0071800 legacy=0x7faff013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 >> v1:172.21.15.71:6789/0 conn(0x7faff0071c40 legacy=0x7faff013b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:54.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 1 monclient: found mon.b 2024-03-25T19:40:54.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff01d31e0 con 0x7faff006ea40 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7fafd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 10 monclient: _renew_subs 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff01d33d0 con 0x7faff006ea40 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 10 monclient: _renew_subs 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faff013c010 con 0x7faff006ea40 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.656+0000 7faff5ad9640 1 librados: init done 2024-03-25T19:40:54.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.657+0000 7faff5ad9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.657+0000 7faff5ad9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.657+0000 7faff5ad9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:54.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.657+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff00627b0 con 0x7faff006ea40 2024-03-25T19:40:54.662 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:09.827 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.678 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:09.828 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.711 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:09.863 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.712 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:09.866 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.783 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:09.945 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.791 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:09.955 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.853 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:10.014 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.874 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:10.021 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.898+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafe0001d30 con 0x7faff006ea40 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.898+0000 7fafd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.898+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fafe00037b0 con 0x7faff006ea40 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.898+0000 7fafd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.898+0000 7fafd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.898+0000 7fafd7fff640 10 monclient: dump: 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:40:54.898 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:40:54.899 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:40:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.898+0000 7fafed7fa640 4 set_mon_vals no callback set 2024-03-25T19:40:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.899+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fafe0019800 con 0x7faff006ea40 2024-03-25T19:40:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.899+0000 7fafeeffd640 10 monclient: get_auth_request con 0x7fafdc06e0f0 auth_method 0 2024-03-25T19:40:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.899+0000 7fafd7fff640 10 client.4216.objecter ms_handle_connect 0x7fafdc06e0f0 2024-03-25T19:40:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.902+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fafe0053560 con 0x7faff006ea40 2024-03-25T19:40:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.902+0000 7fafd7fff640 10 client.4216.objecter ms_dispatch 0x7faff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.902+0000 7fafd7fff640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.902+0000 7fafd7fff640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.902+0000 7fafd7fff640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T19:40:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.902+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafe004eea0 con 0x7faff006ea40 2024-03-25T19:40:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.902+0000 7fafd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:54.902+0000 7fafd7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:54.931 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:10.096 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.951 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:10.102 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.978 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:10.131 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:54.999 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:10.151 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.072 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:10.234 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.078+0000 7faff5ad9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.078+0000 7faff5ad9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.078+0000 7faff5ad9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:40:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.078+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faff00b0630 con 0x7faff006ea40 2024-03-25T19:40:55.087 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:10.251 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.093+0000 7fafd7fff640 1 -- 172.21.15.71:0/1417486951 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fafe0050fa0 con 0x7faff006ea40 2024-03-25T19:40:55.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.093+0000 7fafd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:55.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.093+0000 7fafd7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:55.093 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:40:55.093 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 10 librados: watch_flush enter 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 10 librados: watch_flush exit 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 10 client.4216.objecter successfully canceled tick 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 >> v1:172.21.15.71:6800/3911735624 conn(0x7fafdc06e0f0 legacy=0x7fafdc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 10 monclient: shutdown 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 >> v1:172.21.15.193:6789/0 conn(0x7faff006ea40 legacy=0x7faff01d66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 shutdown_connections 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.094+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 >> 172.21.15.71:0/1417486951 conn(0x7faff006b630 msgr2=0x7faff013d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.095+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 shutdown_connections 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.095+0000 7faff5ad9640 1 -- 172.21.15.71:0/1417486951 wait complete. 2024-03-25T19:40:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:40:55.095+0000 7faff5ad9640 1 librados: shutdown 2024-03-25T19:40:55.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:55.118 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:10.279 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.145 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:10.291 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.194 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:10.359 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.219 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:10.369 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.251 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:10.404 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.290 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:10.442 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.362 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:10.524 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.385 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:10.549 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.413 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:10.559 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.462 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:10.627 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.590 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:10.752 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.644 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:10.806 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.668 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:10.830 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.692 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:10.842 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.728 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:10.882 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.746 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:10.896 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.783 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:10.936 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.815 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:10.966 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.888 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:11.040 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.896 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:11.042 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.912 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:11.076 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.952 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:11.098 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.956 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:11.121 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:55.967 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:11.131 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:56.011 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:11.175 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:56.256 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:11.417 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:40:56.345 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:11.506 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.291 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:16.452 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.327 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:16.481 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.393 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:16.543 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.469 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:16.615 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.520 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:16.671 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.586 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:16.749 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.639 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:16.804 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:01.910 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:02.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.000+0000 7f7a51802640 10 monclient: get_monmap_and_config 2024-03-25T19:41:02.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.000+0000 7f7a51802640 10 monclient: build_initial_monmap 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.000+0000 7f7a51802640 10 monclient: monmap: 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:41:02.002113+0000 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:41:02.002113+0000 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:41:02.001 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.000+0000 7f7a51802640 1 Processor -- start 2024-03-25T19:41:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.000+0000 7f7a51802640 1 -- start start 2024-03-25T19:41:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.000+0000 7f7a51802640 10 monclient: init 2024-03-25T19:41:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 10 monclient(hunting): picked mon.noname-a con 0x7f7a4c0b4a80 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 10 monclient(hunting): picked mon.noname-b con 0x7f7a4c13c270 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 10 monclient(hunting): picked mon.noname-c con 0x7f7a4c0a9930 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c0aaef0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c0ab1d0 con 0x7f7a4c0a9930 2024-03-25T19:41:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c13fc80 con 0x7f7a4c13c270 2024-03-25T19:41:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.001+0000 7f7a51802640 10 monclient(hunting): authenticate will time out at 4868.314591s 2024-03-25T19:41:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.002+0000 7f7a4a7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7a4c13c270 0x7f7a4c142280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33674/0 (socket says 172.21.15.71:33674) 2024-03-25T19:41:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.002+0000 7f7a4a7fc640 1 -- 172.21.15.71:0/2473615834 learned_addr learned my addr 172.21.15.71:0/2473615834 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131590668 0 0) 0x7f7a4c0aaef0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a34003520 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410521198 0 0) 0x7f7a4c13fc80 con 0x7f7a4c13c270 2024-03-25T19:41:02.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 10 monclient(hunting): my global_id is 14219 2024-03-25T19:41:02.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.005+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c0aaef0 con 0x7f7a4c13c270 2024-03-25T19:41:02.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.007+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244938837 0 0) 0x7f7a4c0ab1d0 con 0x7f7a4c0a9930 2024-03-25T19:41:02.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.007+0000 7f7a49ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.007+0000 7f7a49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.007+0000 7f7a49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.007+0000 7f7a49ffb640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:41:02.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.007+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c13fc80 con 0x7f7a4c0a9930 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2711106810 0 0) 0x7f7a34003520 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 >> v1:172.21.15.71:6790/0 conn(0x7f7a4c13c270 legacy=0x7f7a4c142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 >> v1:172.21.15.193:6789/0 conn(0x7f7a4c0a9930 legacy=0x7f7a4c142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 1 monclient: found mon.noname-a 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c0a9ec0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7a3c001690 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a49ffb640 10 monclient: dump: 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a51802640 5 monclient: authenticate success, global_id 4209 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.009+0000 7f7a51802640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:02.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a3c0017f0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a49ffb640 1 -- 172.21.15.71:0/2473615834 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7a3c001c00 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a49ffb640 10 monclient: dump: 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a51802640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:02.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a51802640 4 set_mon_vals no callback set 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a51802640 10 monclient: shutdown 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a51802640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a51802640 1 -- 172.21.15.71:0/2473615834 >> v1:172.21.15.71:6789/0 conn(0x7f7a4c0b4a80 legacy=0x7f7a4c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a51802640 1 -- 172.21.15.71:0/2473615834 shutdown_connections 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.011+0000 7f7a51802640 1 -- 172.21.15.71:0/2473615834 >> 172.21.15.71:0/2473615834 conn(0x7f7a4c0615c0 msgr2=0x7f7a4c0619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.012+0000 7f7a51802640 1 -- 172.21.15.71:0/2473615834 shutdown_connections 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.012+0000 7f7a51802640 1 -- 172.21.15.71:0/2473615834 wait complete. 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.012+0000 7f7a51802640 10 monclient: build_initial_monmap 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.012+0000 7f7a51802640 10 monclient: monmap: 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:41:02.018 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.013+0000 7f7a51802640 1 librados: starting msgr at 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.013+0000 7f7a51802640 1 librados: starting objecter 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.013+0000 7f7a51802640 1 Processor -- start 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.013+0000 7f7a51802640 1 -- start start 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.013+0000 7f7a51802640 1 librados: setting wanted keys 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.013+0000 7f7a51802640 1 librados: calling monclient init 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.013+0000 7f7a51802640 10 monclient: init 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:02.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 10 monclient(hunting): picked mon.noname-b con 0x7f7a4c0a9930 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 10 monclient(hunting): picked mon.noname-c con 0x7f7a4c0b4a80 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 10 monclient(hunting): picked mon.noname-a con 0x7f7a4c13c270 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c146380 con 0x7f7a4c13c270 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c06e0b0 con 0x7f7a4c0a9930 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c06e390 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.014+0000 7f7a51802640 10 monclient(hunting): authenticate will time out at 4868.327193s 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.015+0000 7f7a4a7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7a4c0b4a80 0x7f7a4c1df1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33688/0 (socket says 172.21.15.71:33688) 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.015+0000 7f7a4a7fc640 1 -- 172.21.15.71:0/4133148569 learned_addr learned my addr 172.21.15.71:0/4133148569 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.015+0000 7f7a33fff640 10 client.?.objecter ms_handle_connect 0x7f7a4c0b4a80 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.015+0000 7f7a33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.015+0000 7f7a33fff640 10 client.?.objecter ms_handle_connect 0x7f7a4c0a9930 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.015+0000 7f7a33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.016+0000 7f7a33fff640 10 client.?.objecter ms_handle_connect 0x7f7a4c13c270 2024-03-25T19:41:02.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.016+0000 7f7a33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596015021 0 0) 0x7f7a4c06e390 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 10 monclient(hunting): my global_id is 14222 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a38004eb0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112121473 0 0) 0x7f7a4c06e0b0 con 0x7f7a4c0a9930 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.018+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c06e390 con 0x7f7a4c0a9930 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.019+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801577357 0 0) 0x7f7a4c146380 con 0x7f7a4c13c270 2024-03-25T19:41:02.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.019+0000 7f7a33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.019+0000 7f7a33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.019+0000 7f7a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.019+0000 7f7a33fff640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:41:02.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.019+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c06e0b0 con 0x7f7a4c13c270 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.024+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2820199537 0 0) 0x7f7a38004eb0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.024+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4c146380 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.024+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7a400019b0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.024+0000 7f7a33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.024+0000 7f7a33fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.024+0000 7f7a33fff640 10 monclient(hunting): dump: 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:02.025 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.026 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.026 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.026 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:02.026 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:02.026 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:02.026 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:02.026 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:02.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.024+0000 7f7a33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:02.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.025+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 992915188 0 0) 0x7f7a4c06e0b0 con 0x7f7a4c13c270 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.025+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a38004eb0 con 0x7f7a4c13c270 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7a44001c70 con 0x7f7a4c13c270 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 10 monclient(hunting): dump: 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:02.028 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4078329181 0 0) 0x7f7a4c06e390 con 0x7f7a4c0a9930 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4c06e0b0 con 0x7f7a4c0a9930 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7a3c001b00 con 0x7f7a4c0a9930 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.026+0000 7f7a33fff640 10 monclient(hunting): dump: 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:02.029 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:02.030 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:02.030 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:02.030 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4044054548 0 0) 0x7f7a4c146380 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 >> v1:172.21.15.193:6789/0 conn(0x7f7a4c0a9930 legacy=0x7f7a4c06e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 >> v1:172.21.15.71:6789/0 conn(0x7f7a4c13c270 legacy=0x7f7a4c1e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 1 monclient: found mon.c 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c145510 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a51802640 5 monclient: authenticate success, global_id 14222 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.029+0000 7f7a51802640 10 monclient: _renew_subs 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.030+0000 7f7a51802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.030+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c145a30 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.030+0000 7f7a51802640 10 client.14222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.030+0000 7f7a51802640 10 monclient: _renew_subs 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.030+0000 7f7a51802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.030+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c071ff0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.030+0000 7f7a51802640 1 librados: init done 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.031+0000 7f7a51802640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.031+0000 7f7a51802640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.031+0000 7f7a51802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.031+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4c0aa420 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.032+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a400047e0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.032+0000 7f7a33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.032+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7a40004af0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.032+0000 7f7a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.032+0000 7f7a33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.032+0000 7f7a33fff640 10 monclient: dump: 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.035 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:02.036 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:02.036 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:02.036 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:02.036 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:02.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.032+0000 7f7a497fa640 4 set_mon_vals no callback set 2024-03-25T19:41:02.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.074+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f7a4001aac0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.075+0000 7f7a4affd640 10 monclient: get_auth_request con 0x7f7a3806e0c0 auth_method 0 2024-03-25T19:41:02.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.075+0000 7f7a33fff640 10 client.14222.objecter ms_handle_connect 0x7f7a3806e0c0 2024-03-25T19:41:02.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.091+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f7a40054770 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.091+0000 7f7a33fff640 10 client.14222.objecter ms_dispatch 0x7f7a4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:02.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.091+0000 7f7a33fff640 3 client.14222.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:02.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.091+0000 7f7a33fff640 3 client.14222.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:02.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.091+0000 7f7a33fff640 20 client.14222.objecter dump_active .. 0 homeless 2024-03-25T19:41:02.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.404+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a400500b0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.404+0000 7f7a33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.404+0000 7f7a33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:02.577 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:17.738 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.749+0000 7f7a51802640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.749+0000 7f7a51802640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.749+0000 7f7a51802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:02.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.749+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a4c1456a0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.798+0000 7f7a33fff640 1 -- 172.21.15.71:0/4133148569 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f7a400521b0 con 0x7f7a4c0b4a80 2024-03-25T19:41:02.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.798+0000 7f7a33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.798+0000 7f7a33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:02.800 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:41:02.800 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:02.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.800+0000 7f7a51802640 10 librados: watch_flush enter 2024-03-25T19:41:02.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.800+0000 7f7a51802640 10 librados: watch_flush exit 2024-03-25T19:41:02.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.800+0000 7f7a51802640 20 client.14222.objecter shutdown clearing up homeless session... 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.800+0000 7f7a51802640 10 client.14222.objecter successfully canceled tick 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 >> v1:172.21.15.71:6800/3911735624 conn(0x7f7a3806e0c0 legacy=0x7f7a38070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 10 monclient: shutdown 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 >> v1:172.21.15.71:6790/0 conn(0x7f7a4c0b4a80 legacy=0x7f7a4c1df1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 shutdown_connections 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 >> 172.21.15.71:0/4133148569 conn(0x7f7a4c0615c0 msgr2=0x7f7a4c142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 shutdown_connections 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 1 -- 172.21.15.71:0/4133148569 wait complete. 2024-03-25T19:41:02.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:02.801+0000 7f7a51802640 1 librados: shutdown 2024-03-25T19:41:02.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:06.735 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:21.896 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.738 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:21.900 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.972 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:22.123 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.976 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:22.127 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.023 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:22.169 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.027 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:22.173 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.031 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:22.195 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.035 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:22.199 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.079 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:22.232 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.085 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:22.238 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.232 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:22.382 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.235 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:22.386 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.645 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:22.809 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.648 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:22.813 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:08.338 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:23.499 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:08.344 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:23.505 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.618 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:09.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.717+0000 7fc47aa0d640 10 monclient: get_monmap_and_config 2024-03-25T19:41:09.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.717+0000 7fc47aa0d640 10 monclient: build_initial_monmap 2024-03-25T19:41:09.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.717+0000 7fc47aa0d640 10 monclient: monmap: 2024-03-25T19:41:09.717 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:09.717 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:09.718 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:41:09.718280+0000 2024-03-25T19:41:09.718 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:41:09.718280+0000 2024-03-25T19:41:09.718 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:09.718 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:09.718 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:09.718 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:41:09.718 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:41:09.718 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:09.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.718+0000 7fc47aa0d640 1 Processor -- start 2024-03-25T19:41:09.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.718+0000 7fc47aa0d640 1 -- start start 2024-03-25T19:41:09.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.718+0000 7fc47aa0d640 10 monclient: init 2024-03-25T19:41:09.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.718+0000 7fc47aa0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:09.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:09.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4740b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4740b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 10 monclient(hunting): picked mon.noname-b con 0x7fc47414be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740af940 con 0x7fc4740b4d60 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740b0a70 con 0x7fc4740b1730 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740b0d50 con 0x7fc47414be90 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc47aa0d640 10 monclient(hunting): authenticate will time out at 4876.030924s 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc478faa640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fc47414be90 0x7fc47414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51456/0 (socket says 172.21.15.71:51456) 2024-03-25T19:41:09.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.719+0000 7fc478faa640 1 -- 172.21.15.71:0/410517482 learned_addr learned my addr 172.21.15.71:0/410517482 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:09.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487108119 0 0) 0x7fc4740b0a70 con 0x7fc4740b1730 2024-03-25T19:41:09.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c003520 con 0x7fc4740b1730 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199447214 0 0) 0x7fc4740b0d50 con 0x7fc47414be90 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 10 monclient(hunting): my global_id is 14225 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.722+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4740b0a70 con 0x7fc47414be90 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.723+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94536286 0 0) 0x7fc4740af940 con 0x7fc4740b4d60 2024-03-25T19:41:09.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.723+0000 7fc472ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.723+0000 7fc472ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.723+0000 7fc472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.723+0000 7fc472ffd640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:41:09.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.723+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4740b0d50 con 0x7fc4740b4d60 2024-03-25T19:41:09.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2684673341 0 0) 0x7fc45c003520 con 0x7fc4740b1730 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 >> v1:172.21.15.71:6790/0 conn(0x7fc47414be90 legacy=0x7fc47414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 >> v1:172.21.15.71:6789/0 conn(0x7fc4740b4d60 legacy=0x7fc47413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 1 monclient: found mon.noname-c 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740b1030 con 0x7fc4740b1730 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:09.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc468001690 con 0x7fc4740b1730 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc472ffd640 10 monclient: dump: 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:09.729 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc47aa0d640 5 monclient: authenticate success, global_id 4225 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.726+0000 7fc47aa0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4680017f0 con 0x7fc4740b1730 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc472ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc47aa0d640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc47aa0d640 4 set_mon_vals no callback set 2024-03-25T19:41:09.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc472ffd640 1 -- 172.21.15.71:0/410517482 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc468001c00 con 0x7fc4740b1730 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc472ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc472ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc472ffd640 10 monclient: dump: 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:09.731 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:09.732 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc47aa0d640 10 monclient: shutdown 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc47aa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.728+0000 7fc47aa0d640 1 -- 172.21.15.71:0/410517482 >> v1:172.21.15.193:6789/0 conn(0x7fc4740b1730 legacy=0x7fc4740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.729+0000 7fc47aa0d640 1 -- 172.21.15.71:0/410517482 shutdown_connections 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.729+0000 7fc47aa0d640 1 -- 172.21.15.71:0/410517482 >> 172.21.15.71:0/410517482 conn(0x7fc4740617a0 msgr2=0x7fc474061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.729+0000 7fc47aa0d640 1 -- 172.21.15.71:0/410517482 shutdown_connections 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.729+0000 7fc47aa0d640 1 -- 172.21.15.71:0/410517482 wait complete. 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.729+0000 7fc47aa0d640 10 monclient: build_initial_monmap 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.730+0000 7fc47aa0d640 10 monclient: monmap: 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:09.737 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.730+0000 7fc47aa0d640 1 librados: starting msgr at 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.730+0000 7fc47aa0d640 1 librados: starting objecter 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.730+0000 7fc47aa0d640 1 Processor -- start 2024-03-25T19:41:09.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 1 -- start start 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 1 librados: setting wanted keys 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 1 librados: calling monclient init 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 10 monclient: init 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4740b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4740b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.731+0000 7fc47aa0d640 10 monclient(hunting): picked mon.noname-b con 0x7fc47414be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc47aa0d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc474130710 con 0x7fc4740b4d60 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc47aa0d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47413eb90 con 0x7fc47414be90 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc47aa0d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc474145550 con 0x7fc4740b1730 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc47aa0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:09.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc47aa0d640 10 monclient(hunting): authenticate will time out at 4876.043911s 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc4737fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc4740b4d60 0x7fc474142b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:57266/0 (socket says 172.21.15.71:57266) 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc4737fe640 1 -- 172.21.15.71:0/4122268938 learned_addr learned my addr 172.21.15.71:0/4122268938 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc470ff9640 10 client.?.objecter ms_handle_connect 0x7fc4740b4d60 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.732+0000 7fc470ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.733+0000 7fc470ff9640 10 client.?.objecter ms_handle_connect 0x7fc47414be90 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.733+0000 7fc470ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.733+0000 7fc470ff9640 10 client.?.objecter ms_handle_connect 0x7fc4740b1730 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.733+0000 7fc470ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447106883 0 0) 0x7fc474130710 con 0x7fc4740b4d60 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc460004eb0 con 0x7fc4740b4d60 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968067969 0 0) 0x7fc47413eb90 con 0x7fc47414be90 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 10 monclient(hunting): my global_id is 4228 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.735+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc474130710 con 0x7fc47414be90 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.736+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300904873 0 0) 0x7fc474145550 con 0x7fc4740b1730 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.736+0000 7fc470ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.736+0000 7fc470ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.736+0000 7fc470ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.736+0000 7fc470ff9640 10 monclient(hunting): my global_id is 14228 2024-03-25T19:41:09.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.736+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc47413eb90 con 0x7fc4740b1730 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.741+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3567243110 0 0) 0x7fc460004eb0 con 0x7fc4740b4d60 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.741+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc474145550 con 0x7fc4740b4d60 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.741+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc468001bc0 con 0x7fc4740b4d60 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.741+0000 7fc470ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.742+0000 7fc470ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.742+0000 7fc470ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:09.744 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.742+0000 7fc470ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.742+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 655231675 0 0) 0x7fc474130710 con 0x7fc47414be90 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.742+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc460004eb0 con 0x7fc47414be90 2024-03-25T19:41:09.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc46c001c10 con 0x7fc47414be90 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:09.746 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 958408268 0 0) 0x7fc47413eb90 con 0x7fc4740b1730 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc474130710 con 0x7fc4740b1730 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc464001870 con 0x7fc4740b1730 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.743+0000 7fc470ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:09.747 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:09.748 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.748 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.748 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.748 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:09.748 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:09.748 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:09.748 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:09.748 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:09.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4276849211 0 0) 0x7fc474145550 con 0x7fc4740b4d60 2024-03-25T19:41:09.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 >> v1:172.21.15.71:6790/0 conn(0x7fc4740b1730 legacy=0x7fc474141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 >> v1:172.21.15.193:6789/0 conn(0x7fc47414be90 legacy=0x7fc4741e70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:09.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 1 monclient: found mon.a 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc474143900 con 0x7fc4740b4d60 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc470ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.746+0000 7fc47aa0d640 5 monclient: authenticate success, global_id 4218 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.747+0000 7fc47aa0d640 10 monclient: _renew_subs 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.747+0000 7fc47aa0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.747+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc474137a50 con 0x7fc4740b4d60 2024-03-25T19:41:09.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.747+0000 7fc47aa0d640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:09.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.747+0000 7fc47aa0d640 10 monclient: _renew_subs 2024-03-25T19:41:09.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.747+0000 7fc47aa0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:09.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.747+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47413f020 con 0x7fc4740b4d60 2024-03-25T19:41:09.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.747+0000 7fc47aa0d640 1 librados: init done 2024-03-25T19:41:09.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.748+0000 7fc47aa0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:09.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.748+0000 7fc47aa0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:09.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.748+0000 7fc47aa0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:09.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.748+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4740b2270 con 0x7fc4740b4d60 2024-03-25T19:41:09.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.792+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc468001220 con 0x7fc4740b4d60 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.792+0000 7fc470ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.792+0000 7fc4727fc640 4 set_mon_vals no callback set 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.792+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc468003b00 con 0x7fc4740b4d60 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.792+0000 7fc470ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.792+0000 7fc470ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.792+0000 7fc470ff9640 10 monclient: dump: 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.793 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:09.794 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:09.794 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:09.794 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:09.794 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:09.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.792+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fc468003d20 con 0x7fc4740b4d60 2024-03-25T19:41:09.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.793+0000 7fc473fff640 10 monclient: get_auth_request con 0x7fc46006e0f0 auth_method 0 2024-03-25T19:41:09.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.793+0000 7fc470ff9640 10 client.4218.objecter ms_handle_connect 0x7fc46006e0f0 2024-03-25T19:41:09.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.808+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fc468054960 con 0x7fc4740b4d60 2024-03-25T19:41:09.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.808+0000 7fc470ff9640 10 client.4218.objecter ms_dispatch 0x7fc474000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:09.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.808+0000 7fc470ff9640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:09.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.808+0000 7fc470ff9640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:09.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:09.808+0000 7fc470ff9640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T19:41:10.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.068+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4680504b0 con 0x7fc4740b4d60 2024-03-25T19:41:10.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.068+0000 7fc470ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:10.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.068+0000 7fc470ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:10.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.281+0000 7fc47aa0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:10.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.281+0000 7fc47aa0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:10.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.281+0000 7fc47aa0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:10.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.281+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4740627b0 con 0x7fc4740b4d60 2024-03-25T19:41:10.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.344+0000 7fc470ff9640 1 -- 172.21.15.71:0/4122268938 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fc4680523a0 con 0x7fc4740b4d60 2024-03-25T19:41:10.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.344+0000 7fc470ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:10.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.344+0000 7fc470ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:10.344 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:41:10.344 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 10 librados: watch_flush enter 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 10 librados: watch_flush exit 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 10 client.4218.objecter successfully canceled tick 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 >> v1:172.21.15.71:6800/3911735624 conn(0x7fc46006e0f0 legacy=0x7fc4600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 10 monclient: shutdown 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 >> v1:172.21.15.71:6789/0 conn(0x7fc4740b4d60 legacy=0x7fc474142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 shutdown_connections 2024-03-25T19:41:10.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 >> 172.21.15.71:0/4122268938 conn(0x7fc4740617a0 msgr2=0x7fc474137f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:10.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 shutdown_connections 2024-03-25T19:41:10.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 1 -- 172.21.15.71:0/4122268938 wait complete. 2024-03-25T19:41:10.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:10.347+0000 7fc47aa0d640 1 librados: shutdown 2024-03-25T19:41:10.359 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:10.979 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:26.140 35608== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.000 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:26.151 35609== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.523 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:00:26.673 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.799 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:26.962 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.804 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:00:26.950 35663== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.857 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:00:27.010 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.089 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:00:27.254 35644== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.201 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:28.362 35607== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.167 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:17.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.307+0000 7f9a0fa0f640 10 monclient: get_monmap_and_config 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.307+0000 7f9a0fa0f640 10 monclient: build_initial_monmap 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.307+0000 7f9a0fa0f640 10 monclient: monmap: 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:41:17.308153+0000 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:41:17.308153+0000 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:41:17.307 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:17.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.308+0000 7f9a0fa0f640 1 Processor -- start 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.308+0000 7f9a0fa0f640 1 -- start start 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.308+0000 7f9a0fa0f640 10 monclient: init 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f9a080b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a080b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a0814be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a080af940 con 0x7f9a080b4d60 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a080b0a70 con 0x7f9a0814be90 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a080b0d50 con 0x7f9a080b1730 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:17.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0fa0f640 10 monclient(hunting): authenticate will time out at 4883.621365s 2024-03-25T19:41:17.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0cfaa640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f9a080b1730 0x7f9a080b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45976/0 (socket says 172.21.15.71:45976) 2024-03-25T19:41:17.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.309+0000 7f9a0cfaa640 1 -- 172.21.15.71:0/1477183438 learned_addr learned my addr 172.21.15.71:0/1477183438 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.313+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181057266 0 0) 0x7f9a080b0a70 con 0x7f9a0814be90 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.313+0000 7f99fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.313+0000 7f99fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.313+0000 7f99fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.313+0000 7f99fffff640 10 monclient(hunting): my global_id is 4231 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.313+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f4003520 con 0x7f9a0814be90 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582698280 0 0) 0x7f9a080af940 con 0x7f9a080b4d60 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:17.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:17.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:41:17.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a080b0a70 con 0x7f9a080b4d60 2024-03-25T19:41:17.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054091993 0 0) 0x7f9a080b0d50 con 0x7f9a080b1730 2024-03-25T19:41:17.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:17.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:17.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:17.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 10 monclient(hunting): my global_id is 14231 2024-03-25T19:41:17.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.314+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a080af940 con 0x7f9a080b1730 2024-03-25T19:41:17.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245791646 0 0) 0x7f99f4003520 con 0x7f9a0814be90 2024-03-25T19:41:17.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 >> v1:172.21.15.71:6790/0 conn(0x7f9a080b1730 legacy=0x7f9a080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 >> v1:172.21.15.71:6789/0 conn(0x7f9a080b4d60 legacy=0x7f9a0813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 1 monclient: found mon.noname-c 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a080b1030 con 0x7f9a0814be90 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 10 monclient: _finish_auth 0 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a00001a10 con 0x7f9a0814be90 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f99fffff640 10 monclient: dump: 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f9a0fa0f640 5 monclient: authenticate success, global_id 4231 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.317+0000 7f9a0fa0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:17.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.319+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a00001c30 con 0x7f9a0814be90 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.319+0000 7f99fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.319+0000 7f99fffff640 1 -- 172.21.15.71:0/1477183438 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a00001fc0 con 0x7f9a0814be90 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.319+0000 7f99fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.319+0000 7f99fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.319+0000 7f99fffff640 10 monclient: dump: 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:17.323 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.319+0000 7f9a0fa0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.319+0000 7f9a0fa0f640 4 set_mon_vals no callback set 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.320+0000 7f9a0fa0f640 10 monclient: shutdown 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.320+0000 7f9a0fa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.320+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/1477183438 >> v1:172.21.15.193:6789/0 conn(0x7f9a0814be90 legacy=0x7f9a0814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.320+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/1477183438 shutdown_connections 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.320+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/1477183438 >> 172.21.15.71:0/1477183438 conn(0x7f9a080617a0 msgr2=0x7f9a08061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.320+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/1477183438 shutdown_connections 2024-03-25T19:41:17.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.320+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/1477183438 wait complete. 2024-03-25T19:41:17.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.321+0000 7f9a0fa0f640 10 monclient: build_initial_monmap 2024-03-25T19:41:17.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.321+0000 7f9a0fa0f640 10 monclient: monmap: 2024-03-25T19:41:17.329 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:17.329 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:17.329 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:41:17.329 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:41:17.329 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:17.329 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.322+0000 7f9a0fa0f640 1 librados: starting msgr at 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.322+0000 7f9a0fa0f640 1 librados: starting objecter 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.322+0000 7f9a0fa0f640 1 Processor -- start 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.322+0000 7f9a0fa0f640 1 -- start start 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.323+0000 7f9a0fa0f640 1 librados: setting wanted keys 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.323+0000 7f9a0fa0f640 1 librados: calling monclient init 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.323+0000 7f9a0fa0f640 10 monclient: init 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:17.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a080b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a080b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f9a0814be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08143350 con 0x7f9a0814be90 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08137470 con 0x7f9a080b1730 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08137750 con 0x7f9a080b4d60 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0fa0f640 10 monclient(hunting): authenticate will time out at 4883.636650s 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0cfaa640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f9a080b4d60 0x7f9a08137b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45988/0 (socket says 172.21.15.71:45988) 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.324+0000 7f9a0cfaa640 1 -- 172.21.15.71:0/3840318609 learned_addr learned my addr 172.21.15.71:0/3840318609 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.325+0000 7f99fdffb640 10 client.?.objecter ms_handle_connect 0x7f9a080b4d60 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.325+0000 7f99fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:17.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.325+0000 7f99fdffb640 10 client.?.objecter ms_handle_connect 0x7f9a080b1730 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.325+0000 7f99fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.325+0000 7f99fdffb640 10 client.?.objecter ms_handle_connect 0x7f9a0814be90 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.325+0000 7f99fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.327+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889944630 0 0) 0x7f9a08137470 con 0x7f9a080b1730 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.327+0000 7f99fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.327+0000 7f99fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.327+0000 7f99fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 10 monclient(hunting): my global_id is 4234 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99ec004eb0 con 0x7f9a080b1730 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552137411 0 0) 0x7f9a08143350 con 0x7f9a0814be90 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:17.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:17.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:41:17.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a08137470 con 0x7f9a0814be90 2024-03-25T19:41:17.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403370146 0 0) 0x7f9a08137750 con 0x7f9a080b4d60 2024-03-25T19:41:17.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.328+0000 7f99fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:17.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.329+0000 7f99fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:17.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.329+0000 7f99fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:17.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.329+0000 7f99fdffb640 10 monclient(hunting): my global_id is 14234 2024-03-25T19:41:17.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.329+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a08143350 con 0x7f9a080b4d60 2024-03-25T19:41:17.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.333+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3853146904 0 0) 0x7f99ec004eb0 con 0x7f9a080b1730 2024-03-25T19:41:17.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.333+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a08137750 con 0x7f9a080b1730 2024-03-25T19:41:17.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.333+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a04001970 con 0x7f9a080b1730 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.333+0000 7f99fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.333+0000 7f99fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.333+0000 7f99fdffb640 10 monclient(hunting): dump: 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:17.335 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:17.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.333+0000 7f99fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:17.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.335+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 497909383 0 0) 0x7f9a08137470 con 0x7f9a0814be90 2024-03-25T19:41:17.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99ec004eb0 con 0x7f9a0814be90 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1613218626 0 0) 0x7f9a08143350 con 0x7f9a080b4d60 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a08137470 con 0x7f9a080b4d60 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f99f80017e0 con 0x7f9a080b4d60 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 10 monclient(hunting): dump: 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:17.340 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a00001da0 con 0x7f9a0814be90 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.336+0000 7f99fdffb640 10 monclient(hunting): dump: 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:17.341 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1946580730 0 0) 0x7f9a08137750 con 0x7f9a080b1730 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 >> v1:172.21.15.71:6790/0 conn(0x7f9a080b4d60 legacy=0x7f9a08137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 >> v1:172.21.15.71:6789/0 conn(0x7f9a0814be90 legacy=0x7f9a081e34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 1 monclient: found mon.b 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08142b50 con 0x7f9a080b1730 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:17.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f99fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f9a0fa0f640 5 monclient: authenticate success, global_id 4234 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f9a0fa0f640 10 monclient: _renew_subs 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f9a0fa0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.337+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a08142e30 con 0x7f9a080b1730 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.338+0000 7f9a0fa0f640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.338+0000 7f9a0fa0f640 10 monclient: _renew_subs 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.338+0000 7f9a0fa0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.338+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a081366a0 con 0x7f9a080b1730 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.338+0000 7f9a0fa0f640 1 librados: init done 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.339+0000 7f9a0fa0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.339+0000 7f9a0fa0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.339+0000 7f9a0fa0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:17.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.339+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a081401f0 con 0x7f9a080b1730 2024-03-25T19:41:17.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.356+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a04001ad0 con 0x7f9a080b1730 2024-03-25T19:41:17.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.356+0000 7f99fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:17.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.356+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a040038c0 con 0x7f9a080b1730 2024-03-25T19:41:17.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.356+0000 7f99ff7fe640 4 set_mon_vals no callback set 2024-03-25T19:41:17.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.356+0000 7f99fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.356+0000 7f99fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:17.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.356+0000 7f99fdffb640 10 monclient: dump: 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:17.357 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:17.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.357+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f9a040198d0 con 0x7f9a080b1730 2024-03-25T19:41:17.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.358+0000 7f9a0cfaa640 10 monclient: get_auth_request con 0x7f99ec06e0b0 auth_method 0 2024-03-25T19:41:17.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.358+0000 7f99fdffb640 10 client.4234.objecter ms_handle_connect 0x7f99ec06e0b0 2024-03-25T19:41:17.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.359+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f9a040535a0 con 0x7f9a080b1730 2024-03-25T19:41:17.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.359+0000 7f99fdffb640 10 client.4234.objecter ms_dispatch 0x7f9a08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:17.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.359+0000 7f99fdffb640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:17.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.359+0000 7f99fdffb640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:17.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.359+0000 7f99fdffb640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-25T19:41:17.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.667+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a0404eee0 con 0x7f9a080b1730 2024-03-25T19:41:17.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.667+0000 7f99fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:17.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.667+0000 7f99fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:17.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.979+0000 7f9a0fa0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:17.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.979+0000 7f9a0fa0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:17.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.979+0000 7f9a0fa0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:17.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:17.979+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a0813feb0 con 0x7f9a080b1730 2024-03-25T19:41:18.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.004+0000 7f99fdffb640 1 -- 172.21.15.71:0/3840318609 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f9a04050fe0 con 0x7f9a080b1730 2024-03-25T19:41:18.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.004+0000 7f99fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:18.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.004+0000 7f99fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:18.004 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:41:18.004 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:18.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.007+0000 7f9a0fa0f640 10 librados: watch_flush enter 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.007+0000 7f9a0fa0f640 10 librados: watch_flush exit 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.007+0000 7f9a0fa0f640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.007+0000 7f9a0fa0f640 10 client.4234.objecter successfully canceled tick 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.007+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 >> v1:172.21.15.71:6800/3911735624 conn(0x7f99ec06e0b0 legacy=0x7f99ec070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.007+0000 7f9a0fa0f640 10 monclient: shutdown 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.007+0000 7f9a0fa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.007+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 >> v1:172.21.15.193:6789/0 conn(0x7f9a080b1730 legacy=0x7f9a08133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.008+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 shutdown_connections 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.008+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 >> 172.21.15.71:0/3840318609 conn(0x7f9a080617a0 msgr2=0x7f9a080b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.008+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 shutdown_connections 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.008+0000 7f9a0fa0f640 1 -- 172.21.15.71:0/3840318609 wait complete. 2024-03-25T19:41:18.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:18.008+0000 7f9a0fa0f640 1 librados: shutdown 2024-03-25T19:41:18.020 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:19.801 INFO:tasks.ceph.osd.6.smithi193.stderr:2024-03-25T19:41:19.799+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:41:19.840 INFO:tasks.ceph.osd.5.smithi193.stderr:2024-03-25T19:41:19.838+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:41:20.202 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-03-25T19:41:20.199+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:41:20.356 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-03-25T19:41:20.354+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:41:20.411 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-03-25T19:41:20.409+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:41:20.604 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-03-25T19:41:20.601+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:41:20.635 INFO:tasks.ceph.osd.4.smithi193.stderr:2024-03-25T19:41:20.633+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:41:22.456 INFO:tasks.ceph.osd.7.smithi193.stderr:2024-03-25T19:41:22.453+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:41:24.826 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:24.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.944+0000 7fa136d58640 10 monclient: get_monmap_and_config 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.944+0000 7fa136d58640 10 monclient: build_initial_monmap 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.944+0000 7fa136d58640 10 monclient: monmap: 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:41:24.945122+0000 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:41:24.945122+0000 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:24.944 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:41:24.945 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:41:24.945 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.945+0000 7fa136d58640 1 Processor -- start 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.945+0000 7fa136d58640 1 -- start start 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.945+0000 7fa136d58640 10 monclient: init 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.945+0000 7fa136d58640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 10 monclient(hunting): picked mon.noname-c con 0x7fa1300a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 10 monclient(hunting): picked mon.noname-b con 0x7fa13013f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 10 monclient(hunting): picked mon.noname-a con 0x7fa1301430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa130142520 con 0x7fa1301430c0 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1300b28a0 con 0x7fa1300a98f0 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1300b2a30 con 0x7fa13013f710 2024-03-25T19:41:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:24.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.946+0000 7fa136d58640 10 monclient(hunting): authenticate will time out at 4891.257926s 2024-03-25T19:41:24.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.956+0000 7fa134af4640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa1300a98f0 0x7fa1300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:45722/0 (socket says 172.21.15.71:45722) 2024-03-25T19:41:24.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.956+0000 7fa134af4640 1 -- 172.21.15.71:0/2522496289 learned_addr learned my addr 172.21.15.71:0/2522496289 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:24.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.964+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102931270 0 0) 0x7fa1300b2a30 con 0x7fa13013f710 2024-03-25T19:41:24.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.964+0000 7fa12f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.964+0000 7fa12f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.964+0000 7fa12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.964+0000 7fa12f7fe640 10 monclient(hunting): my global_id is 14261 2024-03-25T19:41:24.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.964+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa120003520 con 0x7fa13013f710 2024-03-25T19:41:24.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.966+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236692162 0 0) 0x7fa130142520 con 0x7fa1301430c0 2024-03-25T19:41:24.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.966+0000 7fa12f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.966+0000 7fa12f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.966+0000 7fa12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.966+0000 7fa12f7fe640 10 monclient(hunting): my global_id is 4251 2024-03-25T19:41:24.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.966+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1300b2a30 con 0x7fa1301430c0 2024-03-25T19:41:24.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.968+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212719665 0 0) 0x7fa1300b28a0 con 0x7fa1300a98f0 2024-03-25T19:41:24.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.968+0000 7fa12f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.968+0000 7fa12f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.968+0000 7fa12f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.968+0000 7fa12f7fe640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:41:24.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.968+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa130142520 con 0x7fa1300a98f0 2024-03-25T19:41:24.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 830224385 0 0) 0x7fa120003520 con 0x7fa13013f710 2024-03-25T19:41:24.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 >> v1:172.21.15.193:6789/0 conn(0x7fa1300a98f0 legacy=0x7fa1300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 >> v1:172.21.15.71:6789/0 conn(0x7fa1301430c0 legacy=0x7fa13014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:24.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 1 monclient: found mon.noname-b 2024-03-25T19:41:24.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1300b44c0 con 0x7fa13013f710 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa11c001690 con 0x7fa13013f710 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa12f7fe640 10 monclient: dump: 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:24.977 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa136d58640 5 monclient: authenticate success, global_id 14261 2024-03-25T19:41:24.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.976+0000 7fa136d58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:24.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.984+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa11c0017f0 con 0x7fa13013f710 2024-03-25T19:41:24.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.984+0000 7fa12f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:24.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.984+0000 7fa12f7fe640 1 -- 172.21.15.71:0/2522496289 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa11c001c00 con 0x7fa13013f710 2024-03-25T19:41:24.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.984+0000 7fa12f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.984+0000 7fa12f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:24.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.984+0000 7fa12f7fe640 10 monclient: dump: 2024-03-25T19:41:24.984 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:24.984 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.984+0000 7fa136d58640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:24.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.984+0000 7fa136d58640 4 set_mon_vals no callback set 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.985+0000 7fa136d58640 10 monclient: shutdown 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.985+0000 7fa136d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.985+0000 7fa136d58640 1 -- 172.21.15.71:0/2522496289 >> v1:172.21.15.71:6790/0 conn(0x7fa13013f710 legacy=0x7fa1301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.985+0000 7fa136d58640 1 -- 172.21.15.71:0/2522496289 shutdown_connections 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.985+0000 7fa136d58640 1 -- 172.21.15.71:0/2522496289 >> 172.21.15.71:0/2522496289 conn(0x7fa1300b7ea0 msgr2=0x7fa1300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.985+0000 7fa136d58640 1 -- 172.21.15.71:0/2522496289 shutdown_connections 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.985+0000 7fa136d58640 1 -- 172.21.15.71:0/2522496289 wait complete. 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.986+0000 7fa136d58640 10 monclient: build_initial_monmap 2024-03-25T19:41:24.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.986+0000 7fa136d58640 10 monclient: monmap: 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:24.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.987+0000 7fa136d58640 1 librados: starting msgr at 2024-03-25T19:41:24.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.987+0000 7fa136d58640 1 librados: starting objecter 2024-03-25T19:41:24.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.987+0000 7fa136d58640 1 Processor -- start 2024-03-25T19:41:24.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.987+0000 7fa136d58640 1 -- start start 2024-03-25T19:41:24.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.987+0000 7fa136d58640 1 librados: setting wanted keys 2024-03-25T19:41:24.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.987+0000 7fa136d58640 1 librados: calling monclient init 2024-03-25T19:41:24.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.987+0000 7fa136d58640 10 monclient: init 2024-03-25T19:41:24.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:24.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:24.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 10 monclient(hunting): picked mon.noname-a con 0x7fa1300a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:24.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 10 monclient(hunting): picked mon.noname-c con 0x7fa13013f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:24.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 10 monclient(hunting): picked mon.noname-b con 0x7fa1301430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:24.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa130138b20 con 0x7fa1300a98f0 2024-03-25T19:41:24.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa130137a80 con 0x7fa1301430c0 2024-03-25T19:41:24.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa130139880 con 0x7fa13013f710 2024-03-25T19:41:24.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:24.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.988+0000 7fa136d58640 10 monclient(hunting): authenticate will time out at 4891.300433s 2024-03-25T19:41:24.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.999+0000 7fa1352f5640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa1301430c0 0x7fa1301e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:45750/0 (socket says 172.21.15.71:45750) 2024-03-25T19:41:24.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.999+0000 7fa1352f5640 1 -- 172.21.15.71:0/4120601694 learned_addr learned my addr 172.21.15.71:0/4120601694 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:24.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.999+0000 7fa12d7fa640 10 client.?.objecter ms_handle_connect 0x7fa1301430c0 2024-03-25T19:41:24.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:24.999+0000 7fa12d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.002+0000 7fa12d7fa640 10 client.?.objecter ms_handle_connect 0x7fa1300a98f0 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.002+0000 7fa12d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:25.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.005+0000 7fa12d7fa640 10 client.?.objecter ms_handle_connect 0x7fa13013f710 2024-03-25T19:41:25.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.005+0000 7fa12d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:25.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.009+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829918250 0 0) 0x7fa130137a80 con 0x7fa1301430c0 2024-03-25T19:41:25.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.009+0000 7fa12d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:25.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.009+0000 7fa12d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:25.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.009+0000 7fa12d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:25.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.009+0000 7fa12d7fa640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:41:25.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.009+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa114004eb0 con 0x7fa1301430c0 2024-03-25T19:41:25.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.015+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520069983 0 0) 0x7fa130138b20 con 0x7fa1300a98f0 2024-03-25T19:41:25.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.015+0000 7fa12d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:25.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.015+0000 7fa12d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:25.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.015+0000 7fa12d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:25.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.015+0000 7fa12d7fa640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:41:25.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.015+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa130137a80 con 0x7fa1300a98f0 2024-03-25T19:41:25.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.016+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734292837 0 0) 0x7fa130139880 con 0x7fa13013f710 2024-03-25T19:41:25.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.016+0000 7fa12d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:25.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.016+0000 7fa12d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:25.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.016+0000 7fa12d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:25.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.016+0000 7fa12d7fa640 10 monclient(hunting): my global_id is 14264 2024-03-25T19:41:25.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.016+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa130138b20 con 0x7fa13013f710 2024-03-25T19:41:25.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.023+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3894515831 0 0) 0x7fa114004eb0 con 0x7fa1301430c0 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.023+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa130139880 con 0x7fa1301430c0 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.023+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa128001c90 con 0x7fa1301430c0 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.023+0000 7fa12d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.023+0000 7fa12d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.023+0000 7fa12d7fa640 10 monclient(hunting): dump: 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:25.023 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:25.024 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:25.024 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:25.024 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:25.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.023+0000 7fa12d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:25.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.026+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4066734475 0 0) 0x7fa130137a80 con 0x7fa1300a98f0 2024-03-25T19:41:25.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.026+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa114004eb0 con 0x7fa1300a98f0 2024-03-25T19:41:25.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.026+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa124001870 con 0x7fa1300a98f0 2024-03-25T19:41:25.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.026+0000 7fa12d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:25.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.026+0000 7fa12d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:25.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.026+0000 7fa12d7fa640 10 monclient(hunting): dump: 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:25.026 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:25.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.029+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2013566622 0 0) 0x7fa130138b20 con 0x7fa13013f710 2024-03-25T19:41:25.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.029+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa130137a80 con 0x7fa13013f710 2024-03-25T19:41:25.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.029+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa11c001b00 con 0x7fa13013f710 2024-03-25T19:41:25.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.029+0000 7fa12d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:25.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.029+0000 7fa12d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:25.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.029+0000 7fa12d7fa640 10 monclient(hunting): dump: 2024-03-25T19:41:25.029 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:25.029 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:25.030 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:25.030 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:25.030 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:25.030 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:25.030 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:25.030 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:25.030 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:25.030 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:25.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2670589480 0 0) 0x7fa130139880 con 0x7fa1301430c0 2024-03-25T19:41:25.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 >> v1:172.21.15.71:6790/0 conn(0x7fa13013f710 legacy=0x7fa130139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:25.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 >> v1:172.21.15.71:6789/0 conn(0x7fa1300a98f0 legacy=0x7fa130137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 1 monclient: found mon.b 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa130139a10 con 0x7fa1301430c0 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa12d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa136d58640 5 monclient: authenticate success, global_id 4264 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa136d58640 10 monclient: _renew_subs 2024-03-25T19:41:25.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa136d58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:25.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.033+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1301dab00 con 0x7fa1301430c0 2024-03-25T19:41:25.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.034+0000 7fa136d58640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:25.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.034+0000 7fa136d58640 10 monclient: _renew_subs 2024-03-25T19:41:25.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.034+0000 7fa136d58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:25.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.034+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13013b310 con 0x7fa1301430c0 2024-03-25T19:41:25.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.034+0000 7fa136d58640 1 librados: init done 2024-03-25T19:41:25.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.035+0000 7fa136d58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:25.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.035+0000 7fa136d58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:25.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.035+0000 7fa136d58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:25.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.035+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa130074cc0 con 0x7fa1301430c0 2024-03-25T19:41:25.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.036+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa128003970 con 0x7fa1301430c0 2024-03-25T19:41:25.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.036+0000 7fa12d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:25.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.036+0000 7fa12effd640 4 set_mon_vals no callback set 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.057+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa128003d40 con 0x7fa1301430c0 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.057+0000 7fa12d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.057+0000 7fa12d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.057+0000 7fa12d7fa640 10 monclient: dump: 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:25.057 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:25.058 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:25.058 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:25.058 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:25.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.058+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fa128003f20 con 0x7fa1301430c0 2024-03-25T19:41:25.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.059+0000 7fa134af4640 10 monclient: get_auth_request con 0x7fa11406e110 auth_method 0 2024-03-25T19:41:25.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.059+0000 7fa12d7fa640 10 client.4264.objecter ms_handle_connect 0x7fa11406e110 2024-03-25T19:41:25.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.063+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fa128054b40 con 0x7fa1301430c0 2024-03-25T19:41:25.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.063+0000 7fa12d7fa640 10 client.4264.objecter ms_dispatch 0x7fa130000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:25.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.063+0000 7fa12d7fa640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:25.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.063+0000 7fa12d7fa640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:25.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.063+0000 7fa12d7fa640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T19:41:25.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.330+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa128050650 con 0x7fa1301430c0 2024-03-25T19:41:25.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.330+0000 7fa12d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:25.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.330+0000 7fa12d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:25.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.579+0000 7fa136d58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.579+0000 7fa136d58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.579+0000 7fa136d58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:25.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.579+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1300b3e80 con 0x7fa1301430c0 2024-03-25T19:41:25.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.602+0000 7fa12d7fa640 1 -- 172.21.15.71:0/4120601694 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fa128052580 con 0x7fa1301430c0 2024-03-25T19:41:25.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.602+0000 7fa12d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.602+0000 7fa12d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:25.602 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:41:25.602 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 10 librados: watch_flush enter 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 10 librados: watch_flush exit 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 10 client.4264.objecter successfully canceled tick 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 >> v1:172.21.15.71:6800/3911735624 conn(0x7fa11406e110 legacy=0x7fa1140705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 10 monclient: shutdown 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 >> v1:172.21.15.193:6789/0 conn(0x7fa1301430c0 legacy=0x7fa1301e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 shutdown_connections 2024-03-25T19:41:25.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 >> 172.21.15.71:0/4120601694 conn(0x7fa1300b7ea0 msgr2=0x7fa130074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:25.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 shutdown_connections 2024-03-25T19:41:25.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 1 -- 172.21.15.71:0/4120601694 wait complete. 2024-03-25T19:41:25.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:25.605+0000 7fa136d58640 1 librados: shutdown 2024-03-25T19:41:25.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:32.423 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.536+0000 7f082dcbb640 10 monclient: get_monmap_and_config 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.536+0000 7f082dcbb640 10 monclient: build_initial_monmap 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.536+0000 7f082dcbb640 10 monclient: monmap: 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:41:32.537526+0000 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:41:32.537526+0000 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:32.536 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:32.537 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:32.537 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:41:32.537 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:41:32.537 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:32.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.537+0000 7f082dcbb640 1 Processor -- start 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.537+0000 7f082dcbb640 1 -- start start 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.537+0000 7f082dcbb640 10 monclient: init 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 10 monclient(hunting): picked mon.noname-a con 0x7f08280a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 10 monclient(hunting): picked mon.noname-b con 0x7f082813f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 10 monclient(hunting): picked mon.noname-c con 0x7f08281430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0828142520 con 0x7f08280a98f0 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08280b28a0 con 0x7f08281430c0 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08280b2a30 con 0x7f082813f710 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.538+0000 7f082dcbb640 10 monclient(hunting): authenticate will time out at 4898.850506s 2024-03-25T19:41:32.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.545+0000 7f0827fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f08281430c0 0x7f082814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:51820/0 (socket says 172.21.15.71:51820) 2024-03-25T19:41:32.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.545+0000 7f0827fff640 1 -- 172.21.15.71:0/2923590022 learned_addr learned my addr 172.21.15.71:0/2923590022 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:32.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.557+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863717102 0 0) 0x7f08280b28a0 con 0x7f08281430c0 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.557+0000 7f08267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.557+0000 7f08267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.557+0000 7f08267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.557+0000 7f08267fc640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.557+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0810003520 con 0x7f08281430c0 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.565+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835302710 0 0) 0x7f08280b2a30 con 0x7f082813f710 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.565+0000 7f08267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.565+0000 7f08267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.565+0000 7f08267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.565+0000 7f08267fc640 10 monclient(hunting): my global_id is 14267 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.565+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08280b28a0 con 0x7f082813f710 2024-03-25T19:41:32.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3863493615 0 0) 0x7f0810003520 con 0x7f08281430c0 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 >> v1:172.21.15.71:6790/0 conn(0x7f082813f710 legacy=0x7f08281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 >> v1:172.21.15.71:6789/0 conn(0x7f08280a98f0 legacy=0x7f08280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 1 monclient: found mon.noname-c 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08280b44c0 con 0x7f08281430c0 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f0820001a10 con 0x7f08281430c0 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f082dcbb640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f082dcbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f08267fc640 10 monclient: dump: 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.575+0000 7f082dcbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.586+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0820001b70 con 0x7f08281430c0 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.586+0000 7f08267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.586+0000 7f08267fc640 1 -- 172.21.15.71:0/2923590022 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f0820001f80 con 0x7f08281430c0 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.586+0000 7f08267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.586+0000 7f08267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.586+0000 7f08267fc640 10 monclient: dump: 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.586+0000 7f082dcbb640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.586+0000 7f082dcbb640 4 set_mon_vals no callback set 2024-03-25T19:41:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.587+0000 7f082dcbb640 10 monclient: shutdown 2024-03-25T19:41:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.587+0000 7f082dcbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.587+0000 7f082dcbb640 1 -- 172.21.15.71:0/2923590022 >> v1:172.21.15.193:6789/0 conn(0x7f08281430c0 legacy=0x7f082814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.587+0000 7f082dcbb640 1 -- 172.21.15.71:0/2923590022 shutdown_connections 2024-03-25T19:41:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.587+0000 7f082dcbb640 1 -- 172.21.15.71:0/2923590022 >> 172.21.15.71:0/2923590022 conn(0x7f08280b7ea0 msgr2=0x7f08280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.588+0000 7f082dcbb640 1 -- 172.21.15.71:0/2923590022 shutdown_connections 2024-03-25T19:41:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.588+0000 7f082dcbb640 1 -- 172.21.15.71:0/2923590022 wait complete. 2024-03-25T19:41:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.589+0000 7f082dcbb640 10 monclient: build_initial_monmap 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.589+0000 7f082dcbb640 10 monclient: monmap: 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:41:32.589 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.589+0000 7f082dcbb640 1 librados: starting msgr at 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.589+0000 7f082dcbb640 1 librados: starting objecter 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.589+0000 7f082dcbb640 1 Processor -- start 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 1 -- start start 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 1 librados: setting wanted keys 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 1 librados: calling monclient init 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 10 monclient: init 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 10 monclient(hunting): picked mon.noname-a con 0x7f08280a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 10 monclient(hunting): picked mon.noname-b con 0x7f082813f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 10 monclient(hunting): picked mon.noname-c con 0x7f08281430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08281369d0 con 0x7f08280a98f0 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0828133a60 con 0x7f082813f710 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08280ab9e0 con 0x7f08281430c0 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.590+0000 7f082dcbb640 10 monclient(hunting): authenticate will time out at 4898.902783s 2024-03-25T19:41:32.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.608+0000 7f0826ffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f082813f710 0x7f08281362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:51884/0 (socket says 172.21.15.71:51884) 2024-03-25T19:41:32.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.608+0000 7f0826ffd640 1 -- 172.21.15.71:0/2684906527 learned_addr learned my addr 172.21.15.71:0/2684906527 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:32.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.609+0000 7f080ffff640 10 client.?.objecter ms_handle_connect 0x7f082813f710 2024-03-25T19:41:32.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.609+0000 7f080ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:32.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.616+0000 7f080ffff640 10 client.?.objecter ms_handle_connect 0x7f08281430c0 2024-03-25T19:41:32.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.616+0000 7f080ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:32.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.622+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518468979 0 0) 0x7f0828133a60 con 0x7f082813f710 2024-03-25T19:41:32.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.622+0000 7f080ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.622+0000 7f080ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.622+0000 7f080ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.622+0000 7f080ffff640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:41:32.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.622+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0814004eb0 con 0x7f082813f710 2024-03-25T19:41:32.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.627+0000 7f080ffff640 10 client.?.objecter ms_handle_connect 0x7f08280a98f0 2024-03-25T19:41:32.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.627+0000 7f080ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.637+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2732833209 0 0) 0x7f0814004eb0 con 0x7f082813f710 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.637+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0828133a60 con 0x7f082813f710 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.637+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f081c001a10 con 0x7f082813f710 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.637+0000 7f080ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.637+0000 7f080ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.637+0000 7f080ffff640 10 monclient(hunting): dump: 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:32.637 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.638 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.638 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.638 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:32.638 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:32.638 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:32.638 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:32.638 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:32.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.637+0000 7f080ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:32.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.642+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516777565 0 0) 0x7f08280ab9e0 con 0x7f08281430c0 2024-03-25T19:41:32.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.642+0000 7f080ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.642+0000 7f080ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.642+0000 7f080ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.642+0000 7f080ffff640 10 monclient(hunting): my global_id is 14270 2024-03-25T19:41:32.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.642+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0814004eb0 con 0x7f08281430c0 2024-03-25T19:41:32.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.648+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175775474 0 0) 0x7f08281369d0 con 0x7f08280a98f0 2024-03-25T19:41:32.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.648+0000 7f080ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.648+0000 7f080ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.648+0000 7f080ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.648+0000 7f080ffff640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:41:32.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.648+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08280ab9e0 con 0x7f08280a98f0 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.656+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 163472729 0 0) 0x7f0814004eb0 con 0x7f08281430c0 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.656+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08281369d0 con 0x7f08281430c0 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.656+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f0820001e80 con 0x7f08281430c0 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.656+0000 7f080ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.656+0000 7f080ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.656+0000 7f080ffff640 10 monclient(hunting): dump: 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.656 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.657 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:32.657 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:32.657 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:32.657 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:32.657 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:32.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 993471598 0 0) 0x7f0828133a60 con 0x7f082813f710 2024-03-25T19:41:32.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 >> v1:172.21.15.71:6790/0 conn(0x7f08281430c0 legacy=0x7f0828133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 >> v1:172.21.15.71:6789/0 conn(0x7f08280a98f0 legacy=0x7f0828132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 1 monclient: found mon.b 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08280abcc0 con 0x7f082813f710 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 10 monclient: _finish_auth 0 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f080ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f082dcbb640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f082dcbb640 10 monclient: _renew_subs 2024-03-25T19:41:32.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f082dcbb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:32.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.664+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08280abfa0 con 0x7f082813f710 2024-03-25T19:41:32.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.665+0000 7f082dcbb640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:32.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.665+0000 7f082dcbb640 10 monclient: _renew_subs 2024-03-25T19:41:32.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.665+0000 7f082dcbb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:32.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.665+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0828137e40 con 0x7f082813f710 2024-03-25T19:41:32.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.665+0000 7f082dcbb640 1 librados: init done 2024-03-25T19:41:32.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.666+0000 7f082dcbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:32.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.666+0000 7f082dcbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:32.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.666+0000 7f082dcbb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:32.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.666+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08280627b0 con 0x7f082813f710 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.955+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f081c001c90 con 0x7f082813f710 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.955+0000 7f080ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.955+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f081c003a60 con 0x7f082813f710 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.955+0000 7f080ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.955+0000 7f080ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.955+0000 7f080ffff640 10 monclient: dump: 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.955 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:32.956 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:32.956 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:32.956 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:32.956 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:32.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.955+0000 7f0825ffb640 4 set_mon_vals no callback set 2024-03-25T19:41:32.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.955+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f081c003c40 con 0x7f082813f710 2024-03-25T19:41:32.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.956+0000 7f08277fe640 10 monclient: get_auth_request con 0x7f081406e020 auth_method 0 2024-03-25T19:41:32.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.956+0000 7f080ffff640 10 client.4270.objecter ms_handle_connect 0x7f081406e020 2024-03-25T19:41:32.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.960+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f081c054810 con 0x7f082813f710 2024-03-25T19:41:32.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.960+0000 7f080ffff640 10 client.4270.objecter ms_dispatch 0x7f0828000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:32.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.960+0000 7f080ffff640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:32.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.960+0000 7f080ffff640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:32.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.960+0000 7f080ffff640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T19:41:32.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.960+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f081c050320 con 0x7f082813f710 2024-03-25T19:41:32.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.960+0000 7f080ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:32.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:32.960+0000 7f080ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:33.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.176+0000 7f082dcbb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:33.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.176+0000 7f082dcbb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:33.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.176+0000 7f082dcbb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:41:33.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.176+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0828139000 con 0x7f082813f710 2024-03-25T19:41:33.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.236+0000 7f080ffff640 1 -- 172.21.15.71:0/2684906527 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f081c052250 con 0x7f082813f710 2024-03-25T19:41:33.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.236+0000 7f080ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:33.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.236+0000 7f080ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:33.236 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:41:33.236 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:33.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 10 librados: watch_flush enter 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 10 librados: watch_flush exit 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 10 client.4270.objecter successfully canceled tick 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 >> v1:172.21.15.71:6800/3911735624 conn(0x7f081406e020 legacy=0x7f08140704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 10 monclient: shutdown 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 >> v1:172.21.15.193:6789/0 conn(0x7f082813f710 legacy=0x7f08281362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 shutdown_connections 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.238+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 >> 172.21.15.71:0/2684906527 conn(0x7f08280b7ea0 msgr2=0x7f0828074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:33.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.239+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 shutdown_connections 2024-03-25T19:41:33.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.239+0000 7f082dcbb640 1 -- 172.21.15.71:0/2684906527 wait complete. 2024-03-25T19:41:33.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:33.239+0000 7f082dcbb640 1 librados: shutdown 2024-03-25T19:41:33.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:36.329 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:19.085 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.390 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:19.147 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.486 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:19.242 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.532 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:19.251 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.560 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:19.317 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.592 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:19.312 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.634 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:19.391 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.789 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:19.545 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:36.823 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:19.542 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:40.055 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:40.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.174+0000 7f3db6752640 10 monclient: get_monmap_and_config 2024-03-25T19:41:40.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.174+0000 7f3db6752640 10 monclient: build_initial_monmap 2024-03-25T19:41:40.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.174+0000 7f3db6752640 10 monclient: monmap: 2024-03-25T19:41:40.174 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:40.174 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:40.174 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:41:40.175557+0000 2024-03-25T19:41:40.174 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:41:40.175557+0000 2024-03-25T19:41:40.175 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:40.175 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:40.175 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:40.175 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:41:40.175 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:41:40.175 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.175+0000 7f3db6752640 1 Processor -- start 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.175+0000 7f3db6752640 1 -- start start 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.175+0000 7f3db6752640 10 monclient: init 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 10 monclient(hunting): picked mon.noname-a con 0x7f3db00a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 10 monclient(hunting): picked mon.noname-b con 0x7f3db013f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 10 monclient(hunting): picked mon.noname-c con 0x7f3db01430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db0142520 con 0x7f3db00a98f0 2024-03-25T19:41:40.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db00b28a0 con 0x7f3db01430c0 2024-03-25T19:41:40.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db00b2a30 con 0x7f3db013f710 2024-03-25T19:41:40.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:40.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.176+0000 7f3db6752640 10 monclient(hunting): authenticate will time out at 4906.488189s 2024-03-25T19:41:40.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.185+0000 7f3daffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3db00a98f0 0x7f3db00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49342/0 (socket says 172.21.15.71:49342) 2024-03-25T19:41:40.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.185+0000 7f3daffff640 1 -- 172.21.15.71:0/3315069451 learned_addr learned my addr 172.21.15.71:0/3315069451 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:40.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.205+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94709122 0 0) 0x7f3db0142520 con 0x7f3db00a98f0 2024-03-25T19:41:40.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.205+0000 7f3daeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.205+0000 7f3daeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.205+0000 7f3daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.205+0000 7f3daeffd640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:41:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.205+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d98003520 con 0x7f3db00a98f0 2024-03-25T19:41:40.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.208+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336355357 0 0) 0x7f3db00b28a0 con 0x7f3db01430c0 2024-03-25T19:41:40.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.208+0000 7f3daeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.208+0000 7f3daeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.208+0000 7f3daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.208+0000 7f3daeffd640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:41:40.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.208+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db0142520 con 0x7f3db01430c0 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.212+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139729865 0 0) 0x7f3db00b2a30 con 0x7f3db013f710 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.212+0000 7f3daeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.212+0000 7f3daeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.212+0000 7f3daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.212+0000 7f3daeffd640 10 monclient(hunting): my global_id is 14273 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.212+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db00b28a0 con 0x7f3db013f710 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007779267 0 0) 0x7f3db0142520 con 0x7f3db01430c0 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 >> v1:172.21.15.71:6790/0 conn(0x7f3db013f710 legacy=0x7f3db01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 >> v1:172.21.15.71:6789/0 conn(0x7f3db00a98f0 legacy=0x7f3db00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 1 monclient: found mon.noname-c 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db00b44c0 con 0x7f3db01430c0 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3da80019f0 con 0x7f3db01430c0 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3db6752640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3db6752640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3daeffd640 10 monclient: dump: 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:40.226 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:40.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.225+0000 7f3db6752640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.239+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3da8001c10 con 0x7f3db01430c0 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.239+0000 7f3daeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.239+0000 7f3daeffd640 1 -- 172.21.15.71:0/3315069451 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3da8001fa0 con 0x7f3db01430c0 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.239+0000 7f3daeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.239+0000 7f3daeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.239+0000 7f3daeffd640 10 monclient: dump: 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:40.239 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:40.240 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:40.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.239+0000 7f3db6752640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:40.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.239+0000 7f3db6752640 4 set_mon_vals no callback set 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.240+0000 7f3db6752640 10 monclient: shutdown 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.240+0000 7f3db6752640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.240+0000 7f3db6752640 1 -- 172.21.15.71:0/3315069451 >> v1:172.21.15.193:6789/0 conn(0x7f3db01430c0 legacy=0x7f3db014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.240+0000 7f3db6752640 1 -- 172.21.15.71:0/3315069451 shutdown_connections 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.240+0000 7f3db6752640 1 -- 172.21.15.71:0/3315069451 >> 172.21.15.71:0/3315069451 conn(0x7f3db00b7ea0 msgr2=0x7f3db00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.240+0000 7f3db6752640 1 -- 172.21.15.71:0/3315069451 shutdown_connections 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.240+0000 7f3db6752640 1 -- 172.21.15.71:0/3315069451 wait complete. 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 10 monclient: build_initial_monmap 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 10 monclient: monmap: 2024-03-25T19:41:40.241 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 1 librados: starting msgr at 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 1 librados: starting objecter 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 1 Processor -- start 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 1 -- start start 2024-03-25T19:41:40.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 1 librados: setting wanted keys 2024-03-25T19:41:40.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 1 librados: calling monclient init 2024-03-25T19:41:40.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.241+0000 7f3db6752640 10 monclient: init 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 10 monclient(hunting): picked mon.noname-b con 0x7f3db00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 10 monclient(hunting): picked mon.noname-c con 0x7f3db013f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 10 monclient(hunting): picked mon.noname-a con 0x7f3db01430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db0137ce0 con 0x7f3db01430c0 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db01396e0 con 0x7f3db00a98f0 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db01399c0 con 0x7f3db013f710 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.242+0000 7f3db6752640 10 monclient(hunting): authenticate will time out at 4906.554670s 2024-03-25T19:41:40.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.249+0000 7f3db4cef640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3db01430c0 0x7f3db01375d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49370/0 (socket says 172.21.15.71:49370) 2024-03-25T19:41:40.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.249+0000 7f3db4cef640 1 -- 172.21.15.71:0/2861585669 learned_addr learned my addr 172.21.15.71:0/2861585669 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:40.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.250+0000 7f3dacff9640 10 client.?.objecter ms_handle_connect 0x7f3db01430c0 2024-03-25T19:41:40.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.250+0000 7f3dacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:40.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.257+0000 7f3dacff9640 10 client.?.objecter ms_handle_connect 0x7f3db00a98f0 2024-03-25T19:41:40.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.257+0000 7f3dacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:40.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495923717 0 0) 0x7f3db0137ce0 con 0x7f3db01430c0 2024-03-25T19:41:40.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d9c004eb0 con 0x7f3db01430c0 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282978206 0 0) 0x7f3db01396e0 con 0x7f3db00a98f0 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:41:40.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db0137ce0 con 0x7f3db00a98f0 2024-03-25T19:41:40.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 client.?.objecter ms_handle_connect 0x7f3db013f710 2024-03-25T19:41:40.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.268+0000 7f3dacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:40.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.280+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601974326 0 0) 0x7f3db01399c0 con 0x7f3db013f710 2024-03-25T19:41:40.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.280+0000 7f3dacff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.280+0000 7f3dacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.280+0000 7f3dacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.280+0000 7f3dacff9640 10 monclient(hunting): my global_id is 14276 2024-03-25T19:41:40.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.280+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db01396e0 con 0x7f3db013f710 2024-03-25T19:41:40.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.286+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 406949038 0 0) 0x7f3d9c004eb0 con 0x7f3db01430c0 2024-03-25T19:41:40.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.287+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3db01399c0 con 0x7f3db01430c0 2024-03-25T19:41:40.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.287+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3da8001fe0 con 0x7f3db01430c0 2024-03-25T19:41:40.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.287+0000 7f3dacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.287+0000 7f3dacff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:40.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.287+0000 7f3dacff9640 10 monclient(hunting): dump: 2024-03-25T19:41:40.286 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:40.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.287+0000 7f3dacff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.288+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2125570076 0 0) 0x7f3db0137ce0 con 0x7f3db00a98f0 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.288+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d9c004eb0 con 0x7f3db00a98f0 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.288+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3da0001950 con 0x7f3db00a98f0 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.288+0000 7f3dacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.288+0000 7f3dacff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.288+0000 7f3dacff9640 10 monclient(hunting): dump: 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.288 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.289 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:40.289 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:40.289 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:40.289 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:40.289 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.295+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3082212251 0 0) 0x7f3db01399c0 con 0x7f3db01430c0 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 >> v1:172.21.15.71:6790/0 conn(0x7f3db013f710 legacy=0x7f3db0138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 >> v1:172.21.15.193:6789/0 conn(0x7f3db00a98f0 legacy=0x7f3db01da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 1 monclient: found mon.a 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db00b0fb0 con 0x7f3db01430c0 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3dacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:40.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 5 monclient: authenticate success, global_id 4263 2024-03-25T19:41:40.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 10 monclient: _renew_subs 2024-03-25T19:41:40.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:40.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db00afdc0 con 0x7f3db01430c0 2024-03-25T19:41:40.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:40.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 10 monclient: _renew_subs 2024-03-25T19:41:40.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:40.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db00b02f0 con 0x7f3db01430c0 2024-03-25T19:41:40.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.296+0000 7f3db6752640 1 librados: init done 2024-03-25T19:41:40.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.297+0000 7f3db6752640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:40.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.297+0000 7f3db6752640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:40.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.297+0000 7f3db6752640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:40.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.297+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db00627b0 con 0x7f3db01430c0 2024-03-25T19:41:40.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.603+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3da8003c20 con 0x7f3db01430c0 2024-03-25T19:41:40.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.603+0000 7f3dacff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:40.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.603+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3da8003fb0 con 0x7f3db01430c0 2024-03-25T19:41:40.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.603+0000 7f3dacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.603+0000 7f3dacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.603+0000 7f3dacff9640 10 monclient: dump: 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:40.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.603+0000 7f3dae7fc640 4 set_mon_vals no callback set 2024-03-25T19:41:40.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.604+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f3da80041d0 con 0x7f3db01430c0 2024-03-25T19:41:40.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.604+0000 7f3daffff640 10 monclient: get_auth_request con 0x7f3d9c06dec0 auth_method 0 2024-03-25T19:41:40.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.605+0000 7f3dacff9640 10 client.4263.objecter ms_handle_connect 0x7f3d9c06dec0 2024-03-25T19:41:40.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.612+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f3da8054bb0 con 0x7f3db01430c0 2024-03-25T19:41:40.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.612+0000 7f3dacff9640 10 client.4263.objecter ms_dispatch 0x7f3db0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:40.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.612+0000 7f3dacff9640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:40.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.612+0000 7f3dacff9640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:40.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.612+0000 7f3dacff9640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-25T19:41:40.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.612+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3da80506c0 con 0x7f3db01430c0 2024-03-25T19:41:40.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.612+0000 7f3dacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:40.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.612+0000 7f3dacff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:40.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.832+0000 7f3db6752640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:40.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.832+0000 7f3db6752640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:40.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.832+0000 7f3db6752640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:40.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.832+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3db00b71a0 con 0x7f3db01430c0 2024-03-25T19:41:40.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.861+0000 7f3dacff9640 1 -- 172.21.15.71:0/2861585669 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f3da80525f0 con 0x7f3db01430c0 2024-03-25T19:41:40.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.861+0000 7f3dacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:40.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.861+0000 7f3dacff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:40.861 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:41:40.861 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 10 librados: watch_flush enter 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 10 librados: watch_flush exit 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 10 client.4263.objecter successfully canceled tick 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 >> v1:172.21.15.71:6800/3911735624 conn(0x7f3d9c06dec0 legacy=0x7f3d9c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 10 monclient: shutdown 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 >> v1:172.21.15.71:6789/0 conn(0x7f3db01430c0 legacy=0x7f3db01375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 shutdown_connections 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 >> 172.21.15.71:0/2861585669 conn(0x7f3db00b7ea0 msgr2=0x7f3db0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 shutdown_connections 2024-03-25T19:41:40.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 1 -- 172.21.15.71:0/2861585669 wait complete. 2024-03-25T19:41:40.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:40.864+0000 7f3db6752640 1 librados: shutdown 2024-03-25T19:41:40.874 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:47.680 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.790+0000 7f09da6c5640 10 monclient: get_monmap_and_config 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.790+0000 7f09da6c5640 10 monclient: build_initial_monmap 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.790+0000 7f09da6c5640 10 monclient: monmap: 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:41:47.791057+0000 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:41:47.791057+0000 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:47.790 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:47.791 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:47.791 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:41:47.791 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:41:47.791 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:47.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.791+0000 7f09da6c5640 1 Processor -- start 2024-03-25T19:41:47.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.791+0000 7f09da6c5640 1 -- start start 2024-03-25T19:41:47.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.791+0000 7f09da6c5640 10 monclient: init 2024-03-25T19:41:47.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.791+0000 7f09da6c5640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:47.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.791+0000 7f09da6c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:47.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.791+0000 7f09da6c5640 10 monclient(hunting): picked mon.noname-c con 0x7f09d40a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:47.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.791+0000 7f09da6c5640 10 monclient(hunting): picked mon.noname-b con 0x7f09d413f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:47.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.792+0000 7f09da6c5640 10 monclient(hunting): picked mon.noname-a con 0x7f09d41430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:47.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.792+0000 7f09da6c5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4142520 con 0x7f09d41430c0 2024-03-25T19:41:47.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.792+0000 7f09da6c5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40b28a0 con 0x7f09d40a98f0 2024-03-25T19:41:47.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.792+0000 7f09da6c5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40b2a30 con 0x7f09d413f710 2024-03-25T19:41:47.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.792+0000 7f09da6c5640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:47.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.792+0000 7f09da6c5640 10 monclient(hunting): authenticate will time out at 4914.103988s 2024-03-25T19:41:47.795 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.796+0000 7f09d37fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f09d413f710 0x7f09d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:37302/0 (socket says 172.21.15.71:37302) 2024-03-25T19:41:47.795 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.796+0000 7f09d37fe640 1 -- 172.21.15.71:0/847454801 learned_addr learned my addr 172.21.15.71:0/847454801 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.804+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783345540 0 0) 0x7f09d40b2a30 con 0x7f09d413f710 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.805+0000 7f09d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.805+0000 7f09d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.805+0000 7f09d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.805+0000 7f09d2ffd640 10 monclient(hunting): my global_id is 14279 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.805+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09bc003520 con 0x7f09d413f710 2024-03-25T19:41:47.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.811+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018682365 0 0) 0x7f09d40b28a0 con 0x7f09d40a98f0 2024-03-25T19:41:47.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.811+0000 7f09d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.811+0000 7f09d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.811+0000 7f09d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.811+0000 7f09d2ffd640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:41:47.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.811+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d40b2a30 con 0x7f09d40a98f0 2024-03-25T19:41:47.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975139473 0 0) 0x7f09bc003520 con 0x7f09d413f710 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 >> v1:172.21.15.193:6789/0 conn(0x7f09d40a98f0 legacy=0x7f09d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 >> v1:172.21.15.71:6789/0 conn(0x7f09d41430c0 legacy=0x7f09d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40b44c0 con 0x7f09d413f710 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f09c8001690 con 0x7f09d413f710 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:47.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09d2ffd640 10 monclient: dump: 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09da6c5640 5 monclient: authenticate success, global_id 14279 2024-03-25T19:41:47.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.816+0000 7f09da6c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:47.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09c80017f0 con 0x7f09d413f710 2024-03-25T19:41:47.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:47.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09d2ffd640 1 -- 172.21.15.71:0/847454801 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f09c8001c00 con 0x7f09d413f710 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09d2ffd640 10 monclient: dump: 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:47.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09da6c5640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:47.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09da6c5640 4 set_mon_vals no callback set 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.821+0000 7f09da6c5640 10 monclient: shutdown 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.822+0000 7f09da6c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.822+0000 7f09da6c5640 1 -- 172.21.15.71:0/847454801 >> v1:172.21.15.71:6790/0 conn(0x7f09d413f710 legacy=0x7f09d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.822+0000 7f09da6c5640 1 -- 172.21.15.71:0/847454801 shutdown_connections 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.822+0000 7f09da6c5640 1 -- 172.21.15.71:0/847454801 >> 172.21.15.71:0/847454801 conn(0x7f09d40b7ea0 msgr2=0x7f09d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.822+0000 7f09da6c5640 1 -- 172.21.15.71:0/847454801 shutdown_connections 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.822+0000 7f09da6c5640 1 -- 172.21.15.71:0/847454801 wait complete. 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.822+0000 7f09da6c5640 10 monclient: build_initial_monmap 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.822+0000 7f09da6c5640 10 monclient: monmap: 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.823+0000 7f09da6c5640 1 librados: starting msgr at 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.823+0000 7f09da6c5640 1 librados: starting objecter 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.823+0000 7f09da6c5640 1 Processor -- start 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.823+0000 7f09da6c5640 1 -- start start 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.823+0000 7f09da6c5640 1 librados: setting wanted keys 2024-03-25T19:41:47.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.823+0000 7f09da6c5640 1 librados: calling monclient init 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.823+0000 7f09da6c5640 10 monclient: init 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 10 monclient(hunting): picked mon.noname-a con 0x7f09d40a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 10 monclient(hunting): picked mon.noname-b con 0x7f09d413f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 10 monclient(hunting): picked mon.noname-c con 0x7f09d41430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d41370a0 con 0x7f09d40a98f0 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4135090 con 0x7f09d413f710 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d4135220 con 0x7f09d41430c0 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:47.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.824+0000 7f09da6c5640 10 monclient(hunting): authenticate will time out at 4914.136129s 2024-03-25T19:41:47.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.834+0000 7f09d3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f09d40a98f0 0x7f09d41dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34762/0 (socket says 172.21.15.71:34762) 2024-03-25T19:41:47.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.834+0000 7f09d3fff640 1 -- 172.21.15.71:0/3053079306 learned_addr learned my addr 172.21.15.71:0/3053079306 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:47.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.842+0000 7f09d0ff9640 10 client.?.objecter ms_handle_connect 0x7f09d40a98f0 2024-03-25T19:41:47.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.842+0000 7f09d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.845+0000 7f09d0ff9640 10 client.?.objecter ms_handle_connect 0x7f09d413f710 2024-03-25T19:41:47.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.845+0000 7f09d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.848+0000 7f09d0ff9640 10 client.?.objecter ms_handle_connect 0x7f09d41430c0 2024-03-25T19:41:47.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.848+0000 7f09d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.851+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376146600 0 0) 0x7f09d41370a0 con 0x7f09d40a98f0 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.851+0000 7f09d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.851+0000 7f09d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.851+0000 7f09d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.851+0000 7f09d0ff9640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.851+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c0004eb0 con 0x7f09d40a98f0 2024-03-25T19:41:47.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.852+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453758510 0 0) 0x7f09d4135090 con 0x7f09d413f710 2024-03-25T19:41:47.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.852+0000 7f09d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.852+0000 7f09d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.852+0000 7f09d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.852+0000 7f09d0ff9640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:41:47.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.852+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d41370a0 con 0x7f09d413f710 2024-03-25T19:41:47.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2295085782 0 0) 0x7f09c0004eb0 con 0x7f09d40a98f0 2024-03-25T19:41:47.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09d4135090 con 0x7f09d40a98f0 2024-03-25T19:41:47.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f09c4001a10 con 0x7f09d40a98f0 2024-03-25T19:41:47.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:47.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:47.866 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:47.866 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177334156 0 0) 0x7f09d4135220 con 0x7f09d41430c0 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 10 monclient(hunting): my global_id is 14282 2024-03-25T19:41:47.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.866+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c0004eb0 con 0x7f09d41430c0 2024-03-25T19:41:47.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.871+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1795507604 0 0) 0x7f09d41370a0 con 0x7f09d413f710 2024-03-25T19:41:47.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.871+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09d4135220 con 0x7f09d413f710 2024-03-25T19:41:47.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.871+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f09c8001b00 con 0x7f09d413f710 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.871+0000 7f09d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.871+0000 7f09d0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.871+0000 7f09d0ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:47.871 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:47.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.874+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1282086168 0 0) 0x7f09c0004eb0 con 0x7f09d41430c0 2024-03-25T19:41:47.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.874+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09d41370a0 con 0x7f09d41430c0 2024-03-25T19:41:47.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.874+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f09cc001a90 con 0x7f09d41430c0 2024-03-25T19:41:47.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.874+0000 7f09d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.874+0000 7f09d0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:47.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.874+0000 7f09d0ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:47.874 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:47.875 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:47.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3522481682 0 0) 0x7f09d4135090 con 0x7f09d40a98f0 2024-03-25T19:41:47.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 >> v1:172.21.15.71:6790/0 conn(0x7f09d41430c0 legacy=0x7f09d41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 >> v1:172.21.15.193:6789/0 conn(0x7f09d413f710 legacy=0x7f09d41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:47.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 1 monclient: found mon.a 2024-03-25T19:41:47.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d41d5000 con 0x7f09d40a98f0 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09da6c5640 5 monclient: authenticate success, global_id 4269 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09da6c5640 10 monclient: _renew_subs 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09da6c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.879+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d41d52b0 con 0x7f09d40a98f0 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.880+0000 7f09da6c5640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.880+0000 7f09da6c5640 10 monclient: _renew_subs 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.880+0000 7f09da6c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.880+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d4135910 con 0x7f09d40a98f0 2024-03-25T19:41:47.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.880+0000 7f09da6c5640 1 librados: init done 2024-03-25T19:41:47.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.881+0000 7f09da6c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.881+0000 7f09da6c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.881+0000 7f09da6c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:47.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:47.881+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d40627b0 con 0x7f09d40a98f0 2024-03-25T19:41:48.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.181+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09c4001c50 con 0x7f09d40a98f0 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.181+0000 7f09d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.181+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f09c4003a60 con 0x7f09d40a98f0 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.181+0000 7f09d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.181+0000 7f09d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.181+0000 7f09d0ff9640 10 monclient: dump: 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:48.181 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:48.182 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:48.182 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:48.182 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:48.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.181+0000 7f09d27fc640 4 set_mon_vals no callback set 2024-03-25T19:41:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.182+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f09c4003c80 con 0x7f09d40a98f0 2024-03-25T19:41:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.182+0000 7f09d37fe640 10 monclient: get_auth_request con 0x7f09c006e160 auth_method 0 2024-03-25T19:41:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.183+0000 7f09d0ff9640 10 client.4269.objecter ms_handle_connect 0x7f09c006e160 2024-03-25T19:41:48.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.188+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f09c4054860 con 0x7f09d40a98f0 2024-03-25T19:41:48.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.188+0000 7f09d0ff9640 10 client.4269.objecter ms_dispatch 0x7f09d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:48.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.188+0000 7f09d0ff9640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:48.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.188+0000 7f09d0ff9640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:48.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.188+0000 7f09d0ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-25T19:41:48.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.188+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09c4050370 con 0x7f09d40a98f0 2024-03-25T19:41:48.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.188+0000 7f09d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:48.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.188+0000 7f09d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:48.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.443+0000 7f09da6c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:48.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.443+0000 7f09da6c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:48.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.443+0000 7f09da6c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:41:48.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.443+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09d413bfe0 con 0x7f09d40a98f0 2024-03-25T19:41:48.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.474+0000 7f09d0ff9640 1 -- 172.21.15.71:0/3053079306 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f09c40522a0 con 0x7f09d40a98f0 2024-03-25T19:41:48.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.474+0000 7f09d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:48.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.474+0000 7f09d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:48.474 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:41:48.474 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:48.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 10 librados: watch_flush enter 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 10 librados: watch_flush exit 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 10 client.4269.objecter successfully canceled tick 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 >> v1:172.21.15.71:6800/3911735624 conn(0x7f09c006e160 legacy=0x7f09c0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 10 monclient: shutdown 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 >> v1:172.21.15.71:6789/0 conn(0x7f09d40a98f0 legacy=0x7f09d41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 shutdown_connections 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.476+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 >> 172.21.15.71:0/3053079306 conn(0x7f09d40b7ea0 msgr2=0x7f09d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.477+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 shutdown_connections 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.477+0000 7f09da6c5640 1 -- 172.21.15.71:0/3053079306 wait complete. 2024-03-25T19:41:48.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:48.477+0000 7f09da6c5640 1 librados: shutdown 2024-03-25T19:41:48.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:55.295 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:55.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.413+0000 7fa97e982640 10 monclient: get_monmap_and_config 2024-03-25T19:41:55.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.413+0000 7fa97e982640 10 monclient: build_initial_monmap 2024-03-25T19:41:55.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.413+0000 7fa97e982640 10 monclient: monmap: 2024-03-25T19:41:55.412 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:55.412 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:55.412 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:41:55.413953+0000 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:41:55.413953+0000 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 1 Processor -- start 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 1 -- start start 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 10 monclient: init 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 10 monclient(hunting): picked mon.noname-b con 0x7fa978061830 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 10 monclient(hunting): picked mon.noname-c con 0x7fa97813d9f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 10 monclient(hunting): picked mon.noname-a con 0x7fa978143d50 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa978071af0 con 0x7fa978143d50 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9781441c0 con 0x7fa97813d9f0 2024-03-25T19:41:55.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa978072810 con 0x7fa978061830 2024-03-25T19:41:55.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:55.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.414+0000 7fa97e982640 10 monclient(hunting): authenticate will time out at 4921.726720s 2024-03-25T19:41:55.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.424+0000 7fa9777fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa97813d9f0 0x7fa978143640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:46626/0 (socket says 172.21.15.71:46626) 2024-03-25T19:41:55.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.425+0000 7fa9777fe640 1 -- 172.21.15.71:0/4116391812 learned_addr learned my addr 172.21.15.71:0/4116391812 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:55.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.436+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819029888 0 0) 0x7fa9781441c0 con 0x7fa97813d9f0 2024-03-25T19:41:55.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.436+0000 7fa976ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:55.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.436+0000 7fa976ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:55.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.436+0000 7fa976ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:55.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.436+0000 7fa976ffd640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:41:55.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.436+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa960003520 con 0x7fa97813d9f0 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592979384 0 0) 0x7fa960003520 con 0x7fa97813d9f0 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 >> v1:172.21.15.71:6790/0 conn(0x7fa978061830 legacy=0x7fa97813d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 >> v1:172.21.15.71:6789/0 conn(0x7fa978143d50 legacy=0x7fa97814b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 1 monclient: found mon.noname-c 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa978072af0 con 0x7fa97813d9f0 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa96c001690 con 0x7fa97813d9f0 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa976ffd640 10 monclient: dump: 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa97e982640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.441+0000 7fa97e982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:55.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa96c0017f0 con 0x7fa97813d9f0 2024-03-25T19:41:55.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa976ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa976ffd640 1 -- 172.21.15.71:0/4116391812 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa96c001c00 con 0x7fa97813d9f0 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa97e982640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa97e982640 4 set_mon_vals no callback set 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa976ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa976ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa976ffd640 10 monclient: dump: 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:55.449 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:55.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa97e982640 10 monclient: shutdown 2024-03-25T19:41:55.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa97e982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa97e982640 1 -- 172.21.15.71:0/4116391812 >> v1:172.21.15.193:6789/0 conn(0x7fa97813d9f0 legacy=0x7fa978143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa97e982640 1 -- 172.21.15.71:0/4116391812 shutdown_connections 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa97e982640 1 -- 172.21.15.71:0/4116391812 >> 172.21.15.71:0/4116391812 conn(0x7fa9780b11e0 msgr2=0x7fa9780b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.449+0000 7fa97e982640 1 -- 172.21.15.71:0/4116391812 shutdown_connections 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.450+0000 7fa97e982640 1 -- 172.21.15.71:0/4116391812 wait complete. 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.450+0000 7fa97e982640 10 monclient: build_initial_monmap 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.450+0000 7fa97e982640 10 monclient: monmap: 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:41:55.452 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 1 librados: starting msgr at 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 1 librados: starting objecter 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 1 Processor -- start 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 1 -- start start 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 1 librados: setting wanted keys 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 1 librados: calling monclient init 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 10 monclient: init 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.451+0000 7fa97e982640 10 monclient(hunting): picked mon.noname-c con 0x7fa978061830 addr v1:172.21.15.71:6790/0 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.452+0000 7fa97e982640 10 monclient(hunting): picked mon.noname-b con 0x7fa97813d9f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.452+0000 7fa97e982640 10 monclient(hunting): picked mon.noname-a con 0x7fa978143d50 addr v1:172.21.15.71:6789/0 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.452+0000 7fa97e982640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9781402b0 con 0x7fa978143d50 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.452+0000 7fa97e982640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa978140da0 con 0x7fa97813d9f0 2024-03-25T19:41:55.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.452+0000 7fa97e982640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97813f580 con 0x7fa978061830 2024-03-25T19:41:55.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.452+0000 7fa97e982640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:55.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.452+0000 7fa97e982640 10 monclient(hunting): authenticate will time out at 4921.763991s 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.461+0000 7fa977fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa978061830 0x7fa97813ee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42092/0 (socket says 172.21.15.71:42092) 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.461+0000 7fa977fff640 1 -- 172.21.15.71:0/114196345 learned_addr learned my addr 172.21.15.71:0/114196345 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:41:55.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.465+0000 7fa974ff9640 10 client.?.objecter ms_handle_connect 0x7fa97813d9f0 2024-03-25T19:41:55.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.465+0000 7fa974ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:55.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.465+0000 7fa974ff9640 10 client.?.objecter ms_handle_connect 0x7fa978061830 2024-03-25T19:41:55.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.465+0000 7fa974ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:55.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.472+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448115300 0 0) 0x7fa97813f580 con 0x7fa978061830 2024-03-25T19:41:55.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.472+0000 7fa974ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:55.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.472+0000 7fa974ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:55.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.472+0000 7fa974ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:55.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.472+0000 7fa974ff9640 10 monclient(hunting): my global_id is 14288 2024-03-25T19:41:55.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.472+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa964004eb0 con 0x7fa978061830 2024-03-25T19:41:55.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.476+0000 7fa974ff9640 10 client.?.objecter ms_handle_connect 0x7fa978143d50 2024-03-25T19:41:55.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.476+0000 7fa974ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:55.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.492+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407096846 0 0) 0x7fa978140da0 con 0x7fa97813d9f0 2024-03-25T19:41:55.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.492+0000 7fa974ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:55.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.492+0000 7fa974ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:55.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.492+0000 7fa974ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:55.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.492+0000 7fa974ff9640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:41:55.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.492+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa97813f580 con 0x7fa97813d9f0 2024-03-25T19:41:55.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.497+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965908560 0 0) 0x7fa9781402b0 con 0x7fa978143d50 2024-03-25T19:41:55.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.497+0000 7fa974ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:55.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.497+0000 7fa974ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:55.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.497+0000 7fa974ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:55.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.497+0000 7fa974ff9640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:41:55.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.497+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa978140da0 con 0x7fa978143d50 2024-03-25T19:41:55.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.499+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3361086477 0 0) 0x7fa97813f580 con 0x7fa97813d9f0 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.499+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9781402b0 con 0x7fa97813d9f0 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.499+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa96c001b40 con 0x7fa97813d9f0 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.499+0000 7fa974ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.499+0000 7fa974ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.499+0000 7fa974ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:55.499 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:55.500 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:55.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.499+0000 7fa974ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.503+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2735743153 0 0) 0x7fa964004eb0 con 0x7fa978061830 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.503+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa97813f580 con 0x7fa978061830 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.503+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa968001a30 con 0x7fa978061830 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.503+0000 7fa974ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.503+0000 7fa974ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.503+0000 7fa974ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:55.503 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:55.504 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:55.504 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:55.504 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2133657195 0 0) 0x7fa97813f580 con 0x7fa978061830 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 >> v1:172.21.15.193:6789/0 conn(0x7fa97813d9f0 legacy=0x7fa9781338a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 >> v1:172.21.15.71:6789/0 conn(0x7fa978143d50 legacy=0x7fa9781d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 1 monclient: found mon.c 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa978140500 con 0x7fa978061830 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa974ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 5 monclient: authenticate success, global_id 14288 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 10 monclient: _renew_subs 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97813dec0 con 0x7fa978061830 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 10 client.14288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 10 monclient: _renew_subs 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa978133fb0 con 0x7fa978061830 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.522+0000 7fa97e982640 1 librados: init done 2024-03-25T19:41:55.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.523+0000 7fa97e982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:55.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.523+0000 7fa97e982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:55.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.523+0000 7fa97e982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:55.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.523+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa978140c00 con 0x7fa978061830 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.854+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa968001c70 con 0x7fa978061830 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.854+0000 7fa974ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.854+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa9680038c0 con 0x7fa978061830 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.854+0000 7fa974ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.854+0000 7fa974ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.854+0000 7fa974ff9640 10 monclient: dump: 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:41:55.855 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.856 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:41:55.856 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:41:55.856 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:41:55.856 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:41:55.856 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:41:55.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.854+0000 7fa9767fc640 4 set_mon_vals no callback set 2024-03-25T19:41:55.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.855+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fa968003aa0 con 0x7fa978061830 2024-03-25T19:41:55.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.856+0000 7fa9777fe640 10 monclient: get_auth_request con 0x7fa96406dfd0 auth_method 0 2024-03-25T19:41:55.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.856+0000 7fa974ff9640 10 client.14288.objecter ms_handle_connect 0x7fa96406dfd0 2024-03-25T19:41:55.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.860+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fa968054710 con 0x7fa978061830 2024-03-25T19:41:55.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.860+0000 7fa974ff9640 10 client.14288.objecter ms_dispatch 0x7fa978000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:55.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.860+0000 7fa974ff9640 3 client.14288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:55.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.860+0000 7fa974ff9640 3 client.14288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:55.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.860+0000 7fa974ff9640 20 client.14288.objecter dump_active .. 0 homeless 2024-03-25T19:41:55.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.860+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa968050220 con 0x7fa978061830 2024-03-25T19:41:55.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.860+0000 7fa974ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:55.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:55.860+0000 7fa974ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:56.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.099+0000 7fa97e982640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:56.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.099+0000 7fa97e982640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:56.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.099+0000 7fa97e982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:41:56.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.099+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9780627b0 con 0x7fa978061830 2024-03-25T19:41:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.140+0000 7fa974ff9640 1 -- 172.21.15.71:0/114196345 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fa968052150 con 0x7fa978061830 2024-03-25T19:41:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.140+0000 7fa974ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.140+0000 7fa974ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:56.142 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:41:56.142 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.142+0000 7fa97e982640 10 librados: watch_flush enter 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.142+0000 7fa97e982640 10 librados: watch_flush exit 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.142+0000 7fa97e982640 20 client.14288.objecter shutdown clearing up homeless session... 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.142+0000 7fa97e982640 10 client.14288.objecter successfully canceled tick 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 >> v1:172.21.15.71:6800/3911735624 conn(0x7fa96406dfd0 legacy=0x7fa964070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 10 monclient: shutdown 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 >> v1:172.21.15.71:6790/0 conn(0x7fa978061830 legacy=0x7fa97813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 shutdown_connections 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 >> 172.21.15.71:0/114196345 conn(0x7fa9780b11e0 msgr2=0x7fa9780afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 shutdown_connections 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 1 -- 172.21.15.71:0/114196345 wait complete. 2024-03-25T19:41:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:41:56.143+0000 7fa97e982640 1 librados: shutdown 2024-03-25T19:41:56.156 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:56.371 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:39.127 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.382 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:39.139 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.427 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:39.183 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.430 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:39.149 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.431 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:39.187 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.437 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:39.157 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.438 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:39.195 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.488 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:39.208 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:56.499 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:39.256 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:02.963 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.078+0000 7f5e6135d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.079+0000 7f5e6135d640 10 monclient: build_initial_monmap 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.079+0000 7f5e6135d640 10 monclient: monmap: 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:42:03.080936+0000 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:42:03.080936+0000 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:42:03.080 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.079+0000 7f5e6135d640 1 Processor -- start 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.079+0000 7f5e6135d640 1 -- start start 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.079+0000 7f5e6135d640 10 monclient: init 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 10 monclient(hunting): picked mon.noname-c con 0x7f5e5c0b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e5c0b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e5c14be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0af940 con 0x7f5e5c14be90 2024-03-25T19:42:03.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0b0a70 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0b0d50 con 0x7f5e5c0b1730 2024-03-25T19:42:03.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:03.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.080+0000 7f5e6135d640 10 monclient(hunting): authenticate will time out at 4929.393445s 2024-03-25T19:42:03.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.093+0000 7f5e5b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5e5c14be90 0x7f5e5c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:55838/0 (socket says 172.21.15.71:55838) 2024-03-25T19:42:03.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.093+0000 7f5e5b7fe640 1 -- 172.21.15.71:0/3967402119 learned_addr learned my addr 172.21.15.71:0/3967402119 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:03.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.108+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864874862 0 0) 0x7f5e5c0af940 con 0x7f5e5c14be90 2024-03-25T19:42:03.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.108+0000 7f5e59ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.108+0000 7f5e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.108+0000 7f5e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.108+0000 7f5e59ffb640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:42:03.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.108+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e44003520 con 0x7f5e5c14be90 2024-03-25T19:42:03.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.109+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606404742 0 0) 0x7f5e5c0b0a70 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.109+0000 7f5e59ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.109+0000 7f5e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.109+0000 7f5e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.109+0000 7f5e59ffb640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:42:03.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.109+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c0af940 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1516307735 0 0) 0x7f5e44003520 con 0x7f5e5c14be90 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 >> v1:172.21.15.71:6790/0 conn(0x7f5e5c0b1730 legacy=0x7f5e5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 >> v1:172.21.15.193:6789/0 conn(0x7f5e5c0b4d60 legacy=0x7f5e5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 1 monclient: found mon.noname-a 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0b1030 con 0x7f5e5c14be90 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5e54001a10 con 0x7f5e5c14be90 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e59ffb640 10 monclient: dump: 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e6135d640 5 monclient: authenticate success, global_id 4278 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.118+0000 7f5e6135d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.139+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e54001b70 con 0x7f5e5c14be90 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.139+0000 7f5e59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.139+0000 7f5e59ffb640 1 -- 172.21.15.71:0/3967402119 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5e54001f80 con 0x7f5e5c14be90 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.139+0000 7f5e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.139+0000 7f5e59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.139+0000 7f5e59ffb640 10 monclient: dump: 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.139+0000 7f5e6135d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:03.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.139+0000 7f5e6135d640 4 set_mon_vals no callback set 2024-03-25T19:42:03.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.140+0000 7f5e6135d640 10 monclient: shutdown 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.140+0000 7f5e6135d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.140+0000 7f5e6135d640 1 -- 172.21.15.71:0/3967402119 >> v1:172.21.15.71:6789/0 conn(0x7f5e5c14be90 legacy=0x7f5e5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.140+0000 7f5e6135d640 1 -- 172.21.15.71:0/3967402119 shutdown_connections 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.140+0000 7f5e6135d640 1 -- 172.21.15.71:0/3967402119 >> 172.21.15.71:0/3967402119 conn(0x7f5e5c0617a0 msgr2=0x7f5e5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.140+0000 7f5e6135d640 1 -- 172.21.15.71:0/3967402119 shutdown_connections 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.140+0000 7f5e6135d640 1 -- 172.21.15.71:0/3967402119 wait complete. 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 10 monclient: build_initial_monmap 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 10 monclient: monmap: 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:42:03.143 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 1 librados: starting msgr at 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 1 librados: starting objecter 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 1 Processor -- start 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 1 -- start start 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 1 librados: setting wanted keys 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 1 librados: calling monclient init 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.141+0000 7f5e6135d640 10 monclient: init 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 10 monclient(hunting): picked mon.noname-c con 0x7f5e5c0b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e5c0b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e5c14be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c134a00 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c139050 con 0x7f5e5c14be90 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c13a330 con 0x7f5e5c0b1730 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.142+0000 7f5e6135d640 10 monclient(hunting): authenticate will time out at 4929.455103s 2024-03-25T19:42:03.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.157+0000 7f5e5affd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5e5c0b1730 0x7f5e5c13ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:41512/0 (socket says 172.21.15.71:41512) 2024-03-25T19:42:03.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.157+0000 7f5e5affd640 1 -- 172.21.15.71:0/198844141 learned_addr learned my addr 172.21.15.71:0/198844141 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:03.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.163+0000 7f5e437fe640 10 client.?.objecter ms_handle_connect 0x7f5e5c0b1730 2024-03-25T19:42:03.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.163+0000 7f5e437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.163+0000 7f5e437fe640 10 client.?.objecter ms_handle_connect 0x7f5e5c0b4d60 2024-03-25T19:42:03.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.163+0000 7f5e437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.168+0000 7f5e437fe640 10 client.?.objecter ms_handle_connect 0x7f5e5c14be90 2024-03-25T19:42:03.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.168+0000 7f5e437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.177+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77823189 0 0) 0x7f5e5c134a00 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.177+0000 7f5e437fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.177+0000 7f5e437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.177+0000 7f5e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.177+0000 7f5e437fe640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:42:03.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.177+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e48004eb0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.179+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566307735 0 0) 0x7f5e5c13a330 con 0x7f5e5c0b1730 2024-03-25T19:42:03.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.179+0000 7f5e437fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.179+0000 7f5e437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.179+0000 7f5e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.179+0000 7f5e437fe640 10 monclient(hunting): my global_id is 14294 2024-03-25T19:42:03.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.179+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c134a00 con 0x7f5e5c0b1730 2024-03-25T19:42:03.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.183+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529716141 0 0) 0x7f5e5c139050 con 0x7f5e5c14be90 2024-03-25T19:42:03.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.183+0000 7f5e437fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.183+0000 7f5e437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.183+0000 7f5e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.183+0000 7f5e437fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:42:03.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.183+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c13a330 con 0x7f5e5c14be90 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.195+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1900300333 0 0) 0x7f5e48004eb0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.195+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e5c139050 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.195+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5e500018c0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.195+0000 7f5e437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.195+0000 7f5e437fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.195+0000 7f5e437fe640 10 monclient(hunting): dump: 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:03.196 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:03.197 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:03.197 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:03.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.195+0000 7f5e437fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:03.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.202+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1998687101 0 0) 0x7f5e5c134a00 con 0x7f5e5c0b1730 2024-03-25T19:42:03.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.202+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e48004eb0 con 0x7f5e5c0b1730 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.202+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5e4c0019f0 con 0x7f5e5c0b1730 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.202+0000 7f5e437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.202+0000 7f5e437fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.202+0000 7f5e437fe640 10 monclient(hunting): dump: 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:03.203 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.217+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1717084149 0 0) 0x7f5e5c139050 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 >> v1:172.21.15.71:6790/0 conn(0x7f5e5c0b1730 legacy=0x7f5e5c13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 >> v1:172.21.15.193:6789/0 conn(0x7f5e5c14be90 legacy=0x7f5e5c1342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 1 monclient: found mon.a 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c13a610 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:03.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 5 monclient: authenticate success, global_id 4281 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 10 monclient: _renew_subs 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c13a8f0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 10 monclient: _renew_subs 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c143140 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 1 librados: init done 2024-03-25T19:42:03.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:03.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.218+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c0627b0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.525+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e500035f0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.525+0000 7f5e437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.525+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5e50003900 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.525+0000 7f5e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.525+0000 7f5e437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.525+0000 7f5e437fe640 10 monclient: dump: 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:03.525 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:03.526 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:03.526 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:03.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.525+0000 7f5e597fa640 4 set_mon_vals no callback set 2024-03-25T19:42:03.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.526+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f5e500198d0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.526+0000 7f5e5affd640 10 monclient: get_auth_request con 0x7f5e4806df70 auth_method 0 2024-03-25T19:42:03.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.526+0000 7f5e437fe640 10 client.4281.objecter ms_handle_connect 0x7f5e4806df70 2024-03-25T19:42:03.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.532+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f5e50053570 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.532+0000 7f5e437fe640 10 client.4281.objecter ms_dispatch 0x7f5e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:03.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.532+0000 7f5e437fe640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:03.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.532+0000 7f5e437fe640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:03.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.532+0000 7f5e437fe640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-25T19:42:03.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.532+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e5004eeb0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.532+0000 7f5e437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.532+0000 7f5e437fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:03.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.788+0000 7f5e6135d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:03.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.788+0000 7f5e6135d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:03.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.788+0000 7f5e6135d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:03.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.788+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e5c1391e0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.843+0000 7f5e437fe640 1 -- 172.21.15.71:0/198844141 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f5e5c1391e0 con 0x7f5e5c0b4d60 2024-03-25T19:42:03.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.843+0000 7f5e437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:03.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.843+0000 7f5e437fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:03.843 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:42:03.843 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:03.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.845+0000 7f5e6135d640 10 librados: watch_flush enter 2024-03-25T19:42:03.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.845+0000 7f5e6135d640 10 librados: watch_flush exit 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.845+0000 7f5e6135d640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.845+0000 7f5e6135d640 10 client.4281.objecter successfully canceled tick 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.845+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 >> v1:172.21.15.71:6800/3911735624 conn(0x7f5e4806df70 legacy=0x7f5e48070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.845+0000 7f5e6135d640 10 monclient: shutdown 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.845+0000 7f5e6135d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.845+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 >> v1:172.21.15.71:6789/0 conn(0x7f5e5c0b4d60 legacy=0x7f5e5c133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.846+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 shutdown_connections 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.846+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 >> 172.21.15.71:0/198844141 conn(0x7f5e5c0617a0 msgr2=0x7f5e5c0b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.846+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 shutdown_connections 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.846+0000 7f5e6135d640 1 -- 172.21.15.71:0/198844141 wait complete. 2024-03-25T19:42:03.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:03.846+0000 7f5e6135d640 1 librados: shutdown 2024-03-25T19:42:03.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:10.665 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.763+0000 7fd03f8cf640 10 monclient: get_monmap_and_config 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.763+0000 7fd03f8cf640 10 monclient: build_initial_monmap 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.763+0000 7fd03f8cf640 10 monclient: monmap: 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:42:10.764310+0000 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:42:10.764310+0000 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:42:10.763 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:42:10.764 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.763+0000 7fd03f8cf640 1 Processor -- start 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 1 -- start start 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 10 monclient: init 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd0380a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd03813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd0381430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:10.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd038142520 con 0x7fd03813f710 2024-03-25T19:42:10.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0380b28a0 con 0x7fd0380a98f0 2024-03-25T19:42:10.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0380b2a30 con 0x7fd0381430c0 2024-03-25T19:42:10.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:10.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.764+0000 7fd03f8cf640 10 monclient(hunting): authenticate will time out at 4937.076733s 2024-03-25T19:42:10.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.765+0000 7fd03de6c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd0381430c0 0x7fd03814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:58604/0 (socket says 172.21.15.71:58604) 2024-03-25T19:42:10.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.765+0000 7fd03de6c640 1 -- 172.21.15.71:0/3321209314 learned_addr learned my addr 172.21.15.71:0/3321209314 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:10.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.783+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172417774 0 0) 0x7fd038142520 con 0x7fd03813f710 2024-03-25T19:42:10.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.783+0000 7fd02ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.783+0000 7fd02ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.783+0000 7fd02ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.783+0000 7fd02ffff640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:42:10.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.783+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd024003520 con 0x7fd03813f710 2024-03-25T19:42:10.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977085121 0 0) 0x7fd024003520 con 0x7fd03813f710 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 >> v1:172.21.15.71:6790/0 conn(0x7fd0381430c0 legacy=0x7fd03814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 >> v1:172.21.15.193:6789/0 conn(0x7fd0380a98f0 legacy=0x7fd0380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 1 monclient: found mon.noname-a 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0380b44c0 con 0x7fd03813f710 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 10 monclient: _finish_auth 0 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd028001690 con 0x7fd03813f710 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd02ffff640 10 monclient: dump: 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd03f8cf640 5 monclient: authenticate success, global_id 4284 2024-03-25T19:42:10.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.810+0000 7fd03f8cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:10.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd028001860 con 0x7fd03813f710 2024-03-25T19:42:10.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd02ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:10.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd02ffff640 1 -- 172.21.15.71:0/3321209314 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd028001cb0 con 0x7fd03813f710 2024-03-25T19:42:10.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd03f8cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:10.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd03f8cf640 4 set_mon_vals no callback set 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd02ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd02ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd02ffff640 10 monclient: dump: 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:10.820 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:10.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd03f8cf640 10 monclient: shutdown 2024-03-25T19:42:10.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd03f8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:10.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3321209314 >> v1:172.21.15.71:6789/0 conn(0x7fd03813f710 legacy=0x7fd0381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3321209314 shutdown_connections 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3321209314 >> 172.21.15.71:0/3321209314 conn(0x7fd0380b7ea0 msgr2=0x7fd0380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.820+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3321209314 shutdown_connections 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.821+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3321209314 wait complete. 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.821+0000 7fd03f8cf640 10 monclient: build_initial_monmap 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.821+0000 7fd03f8cf640 10 monclient: monmap: 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:10.823 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 librados: starting msgr at 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 librados: starting objecter 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 Processor -- start 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 -- start start 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 librados: setting wanted keys 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 librados: calling monclient init 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 10 monclient: init 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd0380a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:10.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd03813f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:10.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd0381430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:10.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd038138e70 con 0x7fd0381430c0 2024-03-25T19:42:10.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd038139150 con 0x7fd03813f710 2024-03-25T19:42:10.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd038139430 con 0x7fd0380a98f0 2024-03-25T19:42:10.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:10.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.822+0000 7fd03f8cf640 10 monclient(hunting): authenticate will time out at 4937.134777s 2024-03-25T19:42:10.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.835+0000 7fd03de6c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd0381430c0 0x7fd0381e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33858/0 (socket says 172.21.15.71:33858) 2024-03-25T19:42:10.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.835+0000 7fd03de6c640 1 -- 172.21.15.71:0/3157781795 learned_addr learned my addr 172.21.15.71:0/3157781795 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:10.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.841+0000 7fd02dffb640 10 client.?.objecter ms_handle_connect 0x7fd03813f710 2024-03-25T19:42:10.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.841+0000 7fd02dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:10.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.841+0000 7fd02dffb640 10 client.?.objecter ms_handle_connect 0x7fd0381430c0 2024-03-25T19:42:10.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.841+0000 7fd02dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:10.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.849+0000 7fd02dffb640 10 client.?.objecter ms_handle_connect 0x7fd0380a98f0 2024-03-25T19:42:10.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.849+0000 7fd02dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:10.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.851+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920498038 0 0) 0x7fd038139150 con 0x7fd03813f710 2024-03-25T19:42:10.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.851+0000 7fd02dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.851+0000 7fd02dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.851+0000 7fd02dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.851+0000 7fd02dffb640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:42:10.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.851+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd01c004eb0 con 0x7fd03813f710 2024-03-25T19:42:10.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.856+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479619460 0 0) 0x7fd038139430 con 0x7fd0380a98f0 2024-03-25T19:42:10.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.856+0000 7fd02dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.856+0000 7fd02dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.856+0000 7fd02dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.856+0000 7fd02dffb640 10 monclient(hunting): my global_id is 14300 2024-03-25T19:42:10.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.856+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd038139150 con 0x7fd0380a98f0 2024-03-25T19:42:10.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.869+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1425728468 0 0) 0x7fd01c004eb0 con 0x7fd03813f710 2024-03-25T19:42:10.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.870+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd038139430 con 0x7fd03813f710 2024-03-25T19:42:10.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.870+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd028001b70 con 0x7fd03813f710 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.870+0000 7fd02dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.870+0000 7fd02dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.870+0000 7fd02dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:10.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.870+0000 7fd02dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:10.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.872+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776818094 0 0) 0x7fd038138e70 con 0x7fd0381430c0 2024-03-25T19:42:10.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.872+0000 7fd02dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.872+0000 7fd02dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.872+0000 7fd02dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.872+0000 7fd02dffb640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:42:10.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.872+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd01c004eb0 con 0x7fd0381430c0 2024-03-25T19:42:10.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.874+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1245070706 0 0) 0x7fd038139150 con 0x7fd0380a98f0 2024-03-25T19:42:10.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.874+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd038138e70 con 0x7fd0380a98f0 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.874+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd034001810 con 0x7fd0380a98f0 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.874+0000 7fd02dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.874+0000 7fd02dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.874+0000 7fd02dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:10.874 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:10.875 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:10.875 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:10.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3775695170 0 0) 0x7fd038138e70 con 0x7fd0380a98f0 2024-03-25T19:42:10.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 >> v1:172.21.15.193:6789/0 conn(0x7fd03813f710 legacy=0x7fd038137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 >> v1:172.21.15.71:6789/0 conn(0x7fd0381430c0 legacy=0x7fd0381e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 1 monclient: found mon.c 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0380b17d0 con 0x7fd0380a98f0 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd02dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.884+0000 7fd03f8cf640 5 monclient: authenticate success, global_id 14300 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.885+0000 7fd03f8cf640 10 monclient: _renew_subs 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.885+0000 7fd03f8cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.885+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd038137040 con 0x7fd0380a98f0 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.885+0000 7fd03f8cf640 10 client.14300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:10.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.885+0000 7fd03f8cf640 10 monclient: _renew_subs 2024-03-25T19:42:10.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.885+0000 7fd03f8cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:10.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.885+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd038139a30 con 0x7fd0380a98f0 2024-03-25T19:42:10.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.885+0000 7fd03f8cf640 1 librados: init done 2024-03-25T19:42:10.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.886+0000 7fd03f8cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:10.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.886+0000 7fd03f8cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:10.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.886+0000 7fd03f8cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:10.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:10.886+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0380b3ef0 con 0x7fd0380a98f0 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.183+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd034001d00 con 0x7fd0380a98f0 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.183+0000 7fd02dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.183+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd034003790 con 0x7fd0380a98f0 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.183+0000 7fd02dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.183+0000 7fd02dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.183+0000 7fd02dffb640 10 monclient: dump: 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:11.183 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:11.184 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:11.184 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:11.184 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:11.184 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:11.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.183+0000 7fd02f7fe640 4 set_mon_vals no callback set 2024-03-25T19:42:11.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.183+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fd0340197a0 con 0x7fd0380a98f0 2024-03-25T19:42:11.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.184+0000 7fd03ce6a640 10 monclient: get_auth_request con 0x7fd01c06df50 auth_method 0 2024-03-25T19:42:11.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.184+0000 7fd02dffb640 10 client.14300.objecter ms_handle_connect 0x7fd01c06df50 2024-03-25T19:42:11.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.194+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fd034053510 con 0x7fd0380a98f0 2024-03-25T19:42:11.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.194+0000 7fd02dffb640 10 client.14300.objecter ms_dispatch 0x7fd038000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:11.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.194+0000 7fd02dffb640 3 client.14300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:11.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.194+0000 7fd02dffb640 3 client.14300.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:11.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.194+0000 7fd02dffb640 20 client.14300.objecter dump_active .. 0 homeless 2024-03-25T19:42:11.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.194+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd03404ee50 con 0x7fd0380a98f0 2024-03-25T19:42:11.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.194+0000 7fd02dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:11.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.194+0000 7fd02dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:11.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.448+0000 7fd03f8cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:11.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.448+0000 7fd03f8cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:11.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.448+0000 7fd03f8cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:11.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.448+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0380627b0 con 0x7fd0380a98f0 2024-03-25T19:42:11.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.481+0000 7fd02dffb640 1 -- 172.21.15.71:0/3157781795 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fd034050f50 con 0x7fd0380a98f0 2024-03-25T19:42:11.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.481+0000 7fd02dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:11.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.481+0000 7fd02dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:11.481 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:42:11.481 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:11.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 10 librados: watch_flush enter 2024-03-25T19:42:11.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 10 librados: watch_flush exit 2024-03-25T19:42:11.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 20 client.14300.objecter shutdown clearing up homeless session... 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 10 client.14300.objecter successfully canceled tick 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 >> v1:172.21.15.71:6800/3911735624 conn(0x7fd01c06df50 legacy=0x7fd01c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 10 monclient: shutdown 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 >> v1:172.21.15.71:6790/0 conn(0x7fd0380a98f0 legacy=0x7fd03813c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 shutdown_connections 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 >> 172.21.15.71:0/3157781795 conn(0x7fd0380b7ea0 msgr2=0x7fd038074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 shutdown_connections 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.483+0000 7fd03f8cf640 1 -- 172.21.15.71:0/3157781795 wait complete. 2024-03-25T19:42:11.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:11.484+0000 7fd03f8cf640 1 librados: shutdown 2024-03-25T19:42:11.495 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:16.445 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:59.202 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:16.455 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:59.212 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:16.488 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:01:59.244 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:16.501 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:59.257 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:16.508 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:59.264 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:16.510 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:59.230 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:16.514 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:59.234 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:16.535 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:01:59.292 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:16.545 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:59.265 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:18.303 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.422+0000 7f8efe96d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.422+0000 7f8efe96d640 10 monclient: build_initial_monmap 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.422+0000 7f8efe96d640 10 monclient: monmap: 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:42:18.423507+0000 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:42:18.423507+0000 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:18.422 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:42:18.423 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:42:18.423 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.423+0000 7f8efe96d640 1 Processor -- start 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.423+0000 7f8efe96d640 1 -- start start 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.423+0000 7f8efe96d640 10 monclient: init 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef80a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef813f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef81430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef8142520 con 0x7f8ef81430c0 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef80b28a0 con 0x7f8ef813f710 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef80b2a30 con 0x7f8ef80a98f0 2024-03-25T19:42:18.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:18.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.424+0000 7f8efe96d640 10 monclient(hunting): authenticate will time out at 4944.736430s 2024-03-25T19:42:18.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.436+0000 7f8ef7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8ef80a98f0 0x7f8ef80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56448/0 (socket says 172.21.15.71:56448) 2024-03-25T19:42:18.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.436+0000 7f8ef7fff640 1 -- 172.21.15.71:0/1064265873 learned_addr learned my addr 172.21.15.71:0/1064265873 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:18.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.448+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833910228 0 0) 0x7f8ef80b28a0 con 0x7f8ef813f710 2024-03-25T19:42:18.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.448+0000 7f8ef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.448+0000 7f8ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.448+0000 7f8ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.448+0000 7f8ef6ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:42:18.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.448+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003520 con 0x7f8ef813f710 2024-03-25T19:42:18.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.450+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742229739 0 0) 0x7f8ef80b2a30 con 0x7f8ef80a98f0 2024-03-25T19:42:18.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.450+0000 7f8ef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.450+0000 7f8ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.450+0000 7f8ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.450+0000 7f8ef6ffd640 10 monclient(hunting): my global_id is 14303 2024-03-25T19:42:18.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.450+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef80b28a0 con 0x7f8ef80a98f0 2024-03-25T19:42:18.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.456+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603484924 0 0) 0x7f8ef8142520 con 0x7f8ef81430c0 2024-03-25T19:42:18.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.456+0000 7f8ef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.456+0000 7f8ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.456+0000 7f8ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.456+0000 7f8ef6ffd640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:42:18.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.456+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef80b2a30 con 0x7f8ef81430c0 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1805400411 0 0) 0x7f8ee0003520 con 0x7f8ef813f710 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 >> v1:172.21.15.71:6790/0 conn(0x7f8ef80a98f0 legacy=0x7f8ef80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 >> v1:172.21.15.71:6789/0 conn(0x7f8ef81430c0 legacy=0x7f8ef814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef80b44c0 con 0x7f8ef813f710 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:18.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8eec001690 con 0x7f8ef813f710 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8ef6ffd640 10 monclient: dump: 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8efe96d640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:42:18.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.464+0000 7f8efe96d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.480+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8eec0017f0 con 0x7f8ef813f710 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.480+0000 7f8ef6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.480+0000 7f8ef6ffd640 1 -- 172.21.15.71:0/1064265873 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8eec001c00 con 0x7f8ef813f710 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.480+0000 7f8ef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.480+0000 7f8ef6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.480+0000 7f8ef6ffd640 10 monclient: dump: 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:18.480 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.480+0000 7f8efe96d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:18.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.480+0000 7f8efe96d640 4 set_mon_vals no callback set 2024-03-25T19:42:18.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.481+0000 7f8efe96d640 10 monclient: shutdown 2024-03-25T19:42:18.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.481+0000 7f8efe96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:18.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.481+0000 7f8efe96d640 1 -- 172.21.15.71:0/1064265873 >> v1:172.21.15.193:6789/0 conn(0x7f8ef813f710 legacy=0x7f8ef81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.481+0000 7f8efe96d640 1 -- 172.21.15.71:0/1064265873 shutdown_connections 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.481+0000 7f8efe96d640 1 -- 172.21.15.71:0/1064265873 >> 172.21.15.71:0/1064265873 conn(0x7f8ef80b7ea0 msgr2=0x7f8ef80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.481+0000 7f8efe96d640 1 -- 172.21.15.71:0/1064265873 shutdown_connections 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.482+0000 7f8efe96d640 1 -- 172.21.15.71:0/1064265873 wait complete. 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.482+0000 7f8efe96d640 10 monclient: build_initial_monmap 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.482+0000 7f8efe96d640 10 monclient: monmap: 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:18.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.483+0000 7f8efe96d640 1 librados: starting msgr at 2024-03-25T19:42:18.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.483+0000 7f8efe96d640 1 librados: starting objecter 2024-03-25T19:42:18.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.483+0000 7f8efe96d640 1 Processor -- start 2024-03-25T19:42:18.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.483+0000 7f8efe96d640 1 -- start start 2024-03-25T19:42:18.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.483+0000 7f8efe96d640 1 librados: setting wanted keys 2024-03-25T19:42:18.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.483+0000 7f8efe96d640 1 librados: calling monclient init 2024-03-25T19:42:18.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.483+0000 7f8efe96d640 10 monclient: init 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef80a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef813f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef81430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef8137140 con 0x7f8ef81430c0 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef81d7b90 con 0x7f8ef813f710 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef81d7590 con 0x7f8ef80a98f0 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:18.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.484+0000 7f8efe96d640 10 monclient(hunting): authenticate will time out at 4944.796275s 2024-03-25T19:42:18.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.500+0000 7f8ef7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8ef80a98f0 0x7f8ef81394b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56492/0 (socket says 172.21.15.71:56492) 2024-03-25T19:42:18.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.500+0000 7f8ef7fff640 1 -- 172.21.15.71:0/2820269007 learned_addr learned my addr 172.21.15.71:0/2820269007 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:18.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.501+0000 7f8ef4ff9640 10 client.?.objecter ms_handle_connect 0x7f8ef80a98f0 2024-03-25T19:42:18.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.501+0000 7f8ef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:18.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.505+0000 7f8ef4ff9640 10 client.?.objecter ms_handle_connect 0x7f8ef813f710 2024-03-25T19:42:18.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.505+0000 7f8ef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:18.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.517+0000 7f8ef4ff9640 10 client.?.objecter ms_handle_connect 0x7f8ef81430c0 2024-03-25T19:42:18.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:18.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126386542 0 0) 0x7f8ef81d7590 con 0x7f8ef80a98f0 2024-03-25T19:42:18.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 monclient(hunting): my global_id is 14306 2024-03-25T19:42:18.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee4004eb0 con 0x7f8ef80a98f0 2024-03-25T19:42:18.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237978964 0 0) 0x7f8ef81d7b90 con 0x7f8ef813f710 2024-03-25T19:42:18.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:42:18.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.518+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef81d7590 con 0x7f8ef813f710 2024-03-25T19:42:18.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.536+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753351555 0 0) 0x7f8ef8137140 con 0x7f8ef81430c0 2024-03-25T19:42:18.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.536+0000 7f8ef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.536+0000 7f8ef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.536+0000 7f8ef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.536+0000 7f8ef4ff9640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:42:18.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.536+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef81d7b90 con 0x7f8ef81430c0 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.538+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2971788400 0 0) 0x7f8ef81d7590 con 0x7f8ef813f710 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.538+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ef8137140 con 0x7f8ef813f710 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.538+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8eec001b00 con 0x7f8ef813f710 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.538+0000 7f8ef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.538+0000 7f8ef4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.538+0000 7f8ef4ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.538 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:18.539 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:18.539 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:18.539 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:18.539 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:18.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.538+0000 7f8ef4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:18.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.546+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4229516034 0 0) 0x7f8ee4004eb0 con 0x7f8ef80a98f0 2024-03-25T19:42:18.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.546+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ef81d7590 con 0x7f8ef80a98f0 2024-03-25T19:42:18.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.546+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8ee80019f0 con 0x7f8ef80a98f0 2024-03-25T19:42:18.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.546+0000 7f8ef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.546+0000 7f8ef4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:18.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.546+0000 7f8ef4ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:18.546 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:18.547 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:18.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.553+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000572841 0 0) 0x7f8ef81d7b90 con 0x7f8ef81430c0 2024-03-25T19:42:18.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.553+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ee4004eb0 con 0x7f8ef81430c0 2024-03-25T19:42:18.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.555+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8ef0001b90 con 0x7f8ef81430c0 2024-03-25T19:42:18.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.555+0000 7f8ef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.555+0000 7f8ef4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.555+0000 7f8ef4ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:18.555 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 728502092 0 0) 0x7f8ef8137140 con 0x7f8ef813f710 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 >> v1:172.21.15.71:6790/0 conn(0x7f8ef80a98f0 legacy=0x7f8ef81394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 >> v1:172.21.15.71:6789/0 conn(0x7f8ef81430c0 legacy=0x7f8ef81e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 1 monclient: found mon.b 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef81d7720 con 0x7f8ef813f710 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8ef4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:18.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:42:18.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 10 monclient: _renew_subs 2024-03-25T19:42:18.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:18.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef813b7d0 con 0x7f8ef813f710 2024-03-25T19:42:18.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:18.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 10 monclient: _renew_subs 2024-03-25T19:42:18.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:18.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef81d80b0 con 0x7f8ef813f710 2024-03-25T19:42:18.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.563+0000 7f8efe96d640 1 librados: init done 2024-03-25T19:42:18.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.564+0000 7f8efe96d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:18.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.564+0000 7f8efe96d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:18.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.564+0000 7f8efe96d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:18.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.564+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef81d7d20 con 0x7f8ef813f710 2024-03-25T19:42:18.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.841+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8eec001c60 con 0x7f8ef813f710 2024-03-25T19:42:18.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.841+0000 7f8ef4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:18.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.841+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8eec003a40 con 0x7f8ef813f710 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.841+0000 7f8ef4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.841+0000 7f8ef4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.841+0000 7f8ef4ff9640 10 monclient: dump: 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:18.900 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:18.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.841+0000 7f8ef67fc640 4 set_mon_vals no callback set 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.842+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f8eec019a10 con 0x7f8ef813f710 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.843+0000 7f8ef7fff640 10 monclient: get_auth_request con 0x7f8ee406e0a0 auth_method 0 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.843+0000 7f8ef4ff9640 10 client.4303.objecter ms_handle_connect 0x7f8ee406e0a0 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.846+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f8eec0535b0 con 0x7f8ef813f710 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.846+0000 7f8ef4ff9640 10 client.4303.objecter ms_dispatch 0x7f8ef8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.846+0000 7f8ef4ff9640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.846+0000 7f8ef4ff9640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.846+0000 7f8ef4ff9640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.847+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8eec04f0c0 con 0x7f8ef813f710 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.847+0000 7f8ef4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:18.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:18.847+0000 7f8ef4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:19.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.087+0000 7f8efe96d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:19.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.087+0000 7f8efe96d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:19.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.087+0000 7f8efe96d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:19.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.087+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ef80627b0 con 0x7f8ef813f710 2024-03-25T19:42:19.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.114+0000 7f8ef4ff9640 1 -- 172.21.15.71:0/2820269007 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f8eec050ff0 con 0x7f8ef813f710 2024-03-25T19:42:19.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.114+0000 7f8ef4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:19.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.114+0000 7f8ef4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:19.114 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:42:19.114 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 10 librados: watch_flush enter 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 10 librados: watch_flush exit 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 10 client.4303.objecter successfully canceled tick 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 >> v1:172.21.15.71:6800/3911735624 conn(0x7f8ee406e0a0 legacy=0x7f8ee4070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 10 monclient: shutdown 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 >> v1:172.21.15.193:6789/0 conn(0x7f8ef813f710 legacy=0x7f8ef8136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:19.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 shutdown_connections 2024-03-25T19:42:19.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 >> 172.21.15.71:0/2820269007 conn(0x7f8ef80b7ea0 msgr2=0x7f8ef8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:19.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 shutdown_connections 2024-03-25T19:42:19.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 1 -- 172.21.15.71:0/2820269007 wait complete. 2024-03-25T19:42:19.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:19.117+0000 7f8efe96d640 1 librados: shutdown 2024-03-25T19:42:19.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:25.937 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:26.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.052+0000 7f19f014d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:26.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.052+0000 7f19f014d640 10 monclient: build_initial_monmap 2024-03-25T19:42:26.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.052+0000 7f19f014d640 10 monclient: monmap: 2024-03-25T19:42:26.052 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:26.052 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:26.053 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:42:26.053691+0000 2024-03-25T19:42:26.053 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:42:26.053691+0000 2024-03-25T19:42:26.053 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:26.053 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:26.053 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:26.053 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:42:26.053 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:42:26.053 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:26.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.053+0000 7f19f014d640 1 Processor -- start 2024-03-25T19:42:26.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.053+0000 7f19f014d640 1 -- start start 2024-03-25T19:42:26.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.053+0000 7f19f014d640 10 monclient: init 2024-03-25T19:42:26.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:26.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:26.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 10 monclient(hunting): picked mon.noname-c con 0x7f19e80b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:26.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 10 monclient(hunting): picked mon.noname-b con 0x7f19e80b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:26.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 10 monclient(hunting): picked mon.noname-a con 0x7f19e814be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:26.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e80af940 con 0x7f19e814be90 2024-03-25T19:42:26.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e80b0a70 con 0x7f19e80b4d60 2024-03-25T19:42:26.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e80b0d50 con 0x7f19e80b1730 2024-03-25T19:42:26.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:26.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.054+0000 7f19f014d640 10 monclient(hunting): authenticate will time out at 4952.366717s 2024-03-25T19:42:26.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.070+0000 7f19ee6ea640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f19e814be90 0x7f19e814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41454/0 (socket says 172.21.15.71:41454) 2024-03-25T19:42:26.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.070+0000 7f19ee6ea640 1 -- 172.21.15.71:0/3377287058 learned_addr learned my addr 172.21.15.71:0/3377287058 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:26.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.078+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422763786 0 0) 0x7f19e80b0d50 con 0x7f19e80b1730 2024-03-25T19:42:26.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.078+0000 7f19ecee7640 10 monclient(hunting): _negotiate 2024-03-25T19:42:26.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.078+0000 7f19ecee7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:26.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.078+0000 7f19ecee7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:26.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.078+0000 7f19ecee7640 10 monclient(hunting): my global_id is 14309 2024-03-25T19:42:26.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.078+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d4003520 con 0x7f19e80b1730 2024-03-25T19:42:26.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.083+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150110244 0 0) 0x7f19e80af940 con 0x7f19e814be90 2024-03-25T19:42:26.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.084+0000 7f19ecee7640 10 monclient(hunting): _negotiate 2024-03-25T19:42:26.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.084+0000 7f19ecee7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:26.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.084+0000 7f19ecee7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:26.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.084+0000 7f19ecee7640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:42:26.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.084+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e80b0d50 con 0x7f19e814be90 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3876536385 0 0) 0x7f19d4003520 con 0x7f19e80b1730 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 >> v1:172.21.15.193:6789/0 conn(0x7f19e80b4d60 legacy=0x7f19e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 >> v1:172.21.15.71:6789/0 conn(0x7f19e814be90 legacy=0x7f19e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 10 monclient: _finish_hunting 0 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 1 monclient: found mon.noname-b 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e80b1030 con 0x7f19e80b1730 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 10 monclient: _finish_auth 0 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 10 monclient: _check_auth_tickets 2024-03-25T19:42:26.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f19d8001690 con 0x7f19e80b1730 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19ecee7640 10 monclient: dump: 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:26.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19f014d640 5 monclient: authenticate success, global_id 14309 2024-03-25T19:42:26.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.091+0000 7f19f014d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:26.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19d80017f0 con 0x7f19e80b1730 2024-03-25T19:42:26.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19ecee7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19ecee7640 1 -- 172.21.15.71:0/3377287058 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f19d8001c00 con 0x7f19e80b1730 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19f014d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19f014d640 4 set_mon_vals no callback set 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19ecee7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19ecee7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19ecee7640 10 monclient: dump: 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:26.100 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19f014d640 10 monclient: shutdown 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19f014d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19f014d640 1 -- 172.21.15.71:0/3377287058 >> v1:172.21.15.71:6790/0 conn(0x7f19e80b1730 legacy=0x7f19e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19f014d640 1 -- 172.21.15.71:0/3377287058 shutdown_connections 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.100+0000 7f19f014d640 1 -- 172.21.15.71:0/3377287058 >> 172.21.15.71:0/3377287058 conn(0x7f19e80617a0 msgr2=0x7f19e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.101+0000 7f19f014d640 1 -- 172.21.15.71:0/3377287058 shutdown_connections 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.101+0000 7f19f014d640 1 -- 172.21.15.71:0/3377287058 wait complete. 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.101+0000 7f19f014d640 10 monclient: build_initial_monmap 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.101+0000 7f19f014d640 10 monclient: monmap: 2024-03-25T19:42:26.103 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.101+0000 7f19f014d640 1 librados: starting msgr at 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.101+0000 7f19f014d640 1 librados: starting objecter 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 1 Processor -- start 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 1 -- start start 2024-03-25T19:42:26.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 1 librados: setting wanted keys 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 1 librados: calling monclient init 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 10 monclient: init 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 10 monclient(hunting): picked mon.noname-b con 0x7f19e80b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 10 monclient(hunting): picked mon.noname-a con 0x7f19e80b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 10 monclient(hunting): picked mon.noname-c con 0x7f19e814be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e81384d0 con 0x7f19e80b4d60 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e8138040 con 0x7f19e80b1730 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e8141dc0 con 0x7f19e814be90 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:26.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.102+0000 7f19f014d640 10 monclient(hunting): authenticate will time out at 4952.414529s 2024-03-25T19:42:26.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.110+0000 7f19ee6ea640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f19e814be90 0x7f19e81e26d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33044/0 (socket says 172.21.15.71:33044) 2024-03-25T19:42:26.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.110+0000 7f19ee6ea640 1 -- 172.21.15.71:0/2826021083 learned_addr learned my addr 172.21.15.71:0/2826021083 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:26.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.112+0000 7f19de7fc640 10 client.?.objecter ms_handle_connect 0x7f19e814be90 2024-03-25T19:42:26.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.112+0000 7f19de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:26.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.114+0000 7f19de7fc640 10 client.?.objecter ms_handle_connect 0x7f19e80b1730 2024-03-25T19:42:26.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.114+0000 7f19de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:26.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.118+0000 7f19de7fc640 10 client.?.objecter ms_handle_connect 0x7f19e80b4d60 2024-03-25T19:42:26.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.118+0000 7f19de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:26.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.120+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073909124 0 0) 0x7f19e8141dc0 con 0x7f19e814be90 2024-03-25T19:42:26.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.120+0000 7f19de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:26.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.120+0000 7f19de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:26.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.120+0000 7f19de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:26.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.120+0000 7f19de7fc640 10 monclient(hunting): my global_id is 14312 2024-03-25T19:42:26.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.120+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19cc004eb0 con 0x7f19e814be90 2024-03-25T19:42:26.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.125+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545855713 0 0) 0x7f19e80b1030 con 0x7f19e80b4d60 2024-03-25T19:42:26.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.125+0000 7f19de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:26.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.125+0000 7f19de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:26.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.125+0000 7f19de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:26.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.125+0000 7f19de7fc640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:42:26.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.125+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e8141dc0 con 0x7f19e80b4d60 2024-03-25T19:42:26.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.128+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746289272 0 0) 0x7f19e8138040 con 0x7f19e80b1730 2024-03-25T19:42:26.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.128+0000 7f19de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:26.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.128+0000 7f19de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:26.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.128+0000 7f19de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:26.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.128+0000 7f19de7fc640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:42:26.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.128+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e80b1030 con 0x7f19e80b1730 2024-03-25T19:42:26.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.147+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3397051807 0 0) 0x7f19e8141dc0 con 0x7f19e80b4d60 2024-03-25T19:42:26.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.147+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19e8138040 con 0x7f19e80b4d60 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.147+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f19d80019a0 con 0x7f19e80b4d60 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.147+0000 7f19de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.147+0000 7f19de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.147+0000 7f19de7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:26.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.147+0000 7f19de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.150+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1770464790 0 0) 0x7f19cc004eb0 con 0x7f19e814be90 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.150+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19e8141dc0 con 0x7f19e814be90 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.150+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f19e0001c10 con 0x7f19e814be90 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.150+0000 7f19de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.150+0000 7f19de7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.150+0000 7f19de7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.150 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.151 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.151 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:26.151 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:26.151 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:26.151 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:26.151 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:26.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.161+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559184038 0 0) 0x7f19e80b1030 con 0x7f19e80b1730 2024-03-25T19:42:26.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.161+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19cc004eb0 con 0x7f19e80b1730 2024-03-25T19:42:26.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.161+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f19e4001860 con 0x7f19e80b1730 2024-03-25T19:42:26.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.161+0000 7f19de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.161+0000 7f19de7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:26.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.161+0000 7f19de7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:26.161 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:26.161 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:26.162 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.162 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.162 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.162 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:26.162 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:26.162 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:26.162 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:26.162 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:26.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 392047725 0 0) 0x7f19e8141dc0 con 0x7f19e814be90 2024-03-25T19:42:26.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 >> v1:172.21.15.193:6789/0 conn(0x7f19e80b1730 legacy=0x7f19e8145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 >> v1:172.21.15.71:6789/0 conn(0x7f19e80b4d60 legacy=0x7f19e8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:26.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 1 monclient: found mon.c 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e81420a0 con 0x7f19e814be90 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 5 monclient: authenticate success, global_id 14312 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 10 monclient: _renew_subs 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e8137ca0 con 0x7f19e814be90 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 10 client.14312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 10 monclient: _renew_subs 2024-03-25T19:42:26.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:26.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e81424c0 con 0x7f19e814be90 2024-03-25T19:42:26.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.166+0000 7f19f014d640 1 librados: init done 2024-03-25T19:42:26.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.167+0000 7f19f014d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:26.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.167+0000 7f19f014d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:26.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.167+0000 7f19f014d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:26.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.167+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e80627b0 con 0x7f19e814be90 2024-03-25T19:42:26.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.484+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e0004b60 con 0x7f19e814be90 2024-03-25T19:42:26.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.484+0000 7f19de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:26.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.484+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f19e0002100 con 0x7f19e814be90 2024-03-25T19:42:26.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.484+0000 7f19de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.484+0000 7f19de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:26.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.484+0000 7f19de7fc640 10 monclient: dump: 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:26.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.484+0000 7f19dffff640 4 set_mon_vals no callback set 2024-03-25T19:42:26.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.485+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f19e001ad60 con 0x7f19e814be90 2024-03-25T19:42:26.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.486+0000 7f19edee9640 10 monclient: get_auth_request con 0x7f19cc06e110 auth_method 0 2024-03-25T19:42:26.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.486+0000 7f19de7fc640 10 client.14312.objecter ms_handle_connect 0x7f19cc06e110 2024-03-25T19:42:26.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.491+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f19e00549f0 con 0x7f19e814be90 2024-03-25T19:42:26.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.491+0000 7f19de7fc640 10 client.14312.objecter ms_dispatch 0x7f19e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:26.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.491+0000 7f19de7fc640 3 client.14312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:26.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.491+0000 7f19de7fc640 3 client.14312.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:26.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.491+0000 7f19de7fc640 20 client.14312.objecter dump_active .. 0 homeless 2024-03-25T19:42:26.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.491+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19e0050330 con 0x7f19e814be90 2024-03-25T19:42:26.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.491+0000 7f19de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:26.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.491+0000 7f19de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:26.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.735+0000 7f19f014d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:26.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.735+0000 7f19f014d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:26.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.735+0000 7f19f014d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:26.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.735+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19e81381f0 con 0x7f19e814be90 2024-03-25T19:42:26.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.775+0000 7f19de7fc640 1 -- 172.21.15.71:0/2826021083 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f19e81381f0 con 0x7f19e814be90 2024-03-25T19:42:26.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.776+0000 7f19de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:26.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.776+0000 7f19de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:26.776 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:42:26.776 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:26.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 10 librados: watch_flush enter 2024-03-25T19:42:26.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 10 librados: watch_flush exit 2024-03-25T19:42:26.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 20 client.14312.objecter shutdown clearing up homeless session... 2024-03-25T19:42:26.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 10 client.14312.objecter successfully canceled tick 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 >> v1:172.21.15.71:6800/3911735624 conn(0x7f19cc06e110 legacy=0x7f19cc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 10 monclient: shutdown 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 >> v1:172.21.15.71:6790/0 conn(0x7f19e814be90 legacy=0x7f19e81e26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 shutdown_connections 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 >> 172.21.15.71:0/2826021083 conn(0x7f19e80617a0 msgr2=0x7f19e80b6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 shutdown_connections 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 1 -- 172.21.15.71:0/2826021083 wait complete. 2024-03-25T19:42:26.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:26.778+0000 7f19f014d640 1 librados: shutdown 2024-03-25T19:42:26.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:33.596 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.709+0000 7f654c1ef640 10 monclient: get_monmap_and_config 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.709+0000 7f654c1ef640 10 monclient: build_initial_monmap 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.709+0000 7f654c1ef640 10 monclient: monmap: 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:42:33.710374+0000 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:42:33.710374+0000 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:33.709 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:33.710 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:42:33.710 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:42:33.710 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.710+0000 7f654c1ef640 1 Processor -- start 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.710+0000 7f654c1ef640 1 -- start start 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.710+0000 7f654c1ef640 10 monclient: init 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 10 monclient(hunting): picked mon.noname-b con 0x7f65440a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 10 monclient(hunting): picked mon.noname-a con 0x7f654413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 10 monclient(hunting): picked mon.noname-c con 0x7f65441430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6544142520 con 0x7f654413f710 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65440b28a0 con 0x7f65441430c0 2024-03-25T19:42:33.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65440b2a30 con 0x7f65440a98f0 2024-03-25T19:42:33.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:33.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.711+0000 7f654c1ef640 10 monclient(hunting): authenticate will time out at 4960.023197s 2024-03-25T19:42:33.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.714+0000 7f654a78c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f65441430c0 0x7f654414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:59016/0 (socket says 172.21.15.71:59016) 2024-03-25T19:42:33.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.714+0000 7f654a78c640 1 -- 172.21.15.71:0/165417899 learned_addr learned my addr 172.21.15.71:0/165417899 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:33.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.743+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021703092 0 0) 0x7f6544142520 con 0x7f654413f710 2024-03-25T19:42:33.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.743+0000 7f6548f89640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.743+0000 7f6548f89640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.743+0000 7f6548f89640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.743+0000 7f6548f89640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:42:33.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.743+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6530003520 con 0x7f654413f710 2024-03-25T19:42:33.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2574121725 0 0) 0x7f6530003520 con 0x7f654413f710 2024-03-25T19:42:33.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 >> v1:172.21.15.71:6790/0 conn(0x7f65440a98f0 legacy=0x7f65440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 >> v1:172.21.15.193:6789/0 conn(0x7f65441430c0 legacy=0x7f654414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 10 monclient: _finish_hunting 0 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 1 monclient: found mon.noname-a 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65440b44c0 con 0x7f654413f710 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 10 monclient: _finish_auth 0 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 10 monclient: _check_auth_tickets 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f654c1ef640 5 monclient: authenticate success, global_id 4302 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6534001690 con 0x7f654413f710 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f6548f89640 10 monclient: dump: 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:33.766 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:33.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.766+0000 7f654c1ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:33.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.829+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65340017f0 con 0x7f654413f710 2024-03-25T19:42:33.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.829+0000 7f6548f89640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:33.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.829+0000 7f6548f89640 1 -- 172.21.15.71:0/165417899 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6534001c00 con 0x7f654413f710 2024-03-25T19:42:33.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.829+0000 7f6548f89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:33.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.829+0000 7f6548f89640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:33.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.829+0000 7f6548f89640 10 monclient: dump: 2024-03-25T19:42:33.829 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.829+0000 7f654c1ef640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:33.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.829+0000 7f654c1ef640 4 set_mon_vals no callback set 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.830+0000 7f654c1ef640 10 monclient: shutdown 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.830+0000 7f654c1ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.830+0000 7f654c1ef640 1 -- 172.21.15.71:0/165417899 >> v1:172.21.15.71:6789/0 conn(0x7f654413f710 legacy=0x7f65441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.831+0000 7f654c1ef640 1 -- 172.21.15.71:0/165417899 shutdown_connections 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.831+0000 7f654c1ef640 1 -- 172.21.15.71:0/165417899 >> 172.21.15.71:0/165417899 conn(0x7f65440b7ea0 msgr2=0x7f65440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.831+0000 7f654c1ef640 1 -- 172.21.15.71:0/165417899 shutdown_connections 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.831+0000 7f654c1ef640 1 -- 172.21.15.71:0/165417899 wait complete. 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 10 monclient: build_initial_monmap 2024-03-25T19:42:33.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 10 monclient: monmap: 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 1 librados: starting msgr at 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 1 librados: starting objecter 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 1 Processor -- start 2024-03-25T19:42:33.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 1 -- start start 2024-03-25T19:42:33.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 1 librados: setting wanted keys 2024-03-25T19:42:33.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 1 librados: calling monclient init 2024-03-25T19:42:33.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.832+0000 7f654c1ef640 10 monclient: init 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 10 monclient(hunting): picked mon.noname-a con 0x7f65440a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 10 monclient(hunting): picked mon.noname-b con 0x7f654413f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 10 monclient(hunting): picked mon.noname-c con 0x7f65441430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65440abd70 con 0x7f65440a98f0 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65441d7710 con 0x7f654413f710 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65441d8340 con 0x7f65441430c0 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:33.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.833+0000 7f654c1ef640 10 monclient(hunting): authenticate will time out at 4960.145828s 2024-03-25T19:42:33.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.840+0000 7f654a78c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f65441430c0 0x7f65441d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42478/0 (socket says 172.21.15.71:42478) 2024-03-25T19:42:33.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.840+0000 7f654a78c640 1 -- 172.21.15.71:0/588088610 learned_addr learned my addr 172.21.15.71:0/588088610 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:33.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.841+0000 7f653a7fc640 10 client.?.objecter ms_handle_connect 0x7f65441430c0 2024-03-25T19:42:33.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.841+0000 7f653a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:33.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.847+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047460792 0 0) 0x7f65441d8340 con 0x7f65441430c0 2024-03-25T19:42:33.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.848+0000 7f653a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.848+0000 7f653a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.848+0000 7f653a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.848+0000 7f653a7fc640 10 monclient(hunting): my global_id is 14315 2024-03-25T19:42:33.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.848+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6528004eb0 con 0x7f65441430c0 2024-03-25T19:42:33.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.848+0000 7f653a7fc640 10 client.?.objecter ms_handle_connect 0x7f65440a98f0 2024-03-25T19:42:33.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.848+0000 7f653a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:33.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.852+0000 7f653a7fc640 10 client.?.objecter ms_handle_connect 0x7f654413f710 2024-03-25T19:42:33.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.852+0000 7f653a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:33.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.861+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699613323 0 0) 0x7f65441d7710 con 0x7f654413f710 2024-03-25T19:42:33.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.861+0000 7f653a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.861+0000 7f653a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.861+0000 7f653a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.861+0000 7f653a7fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:42:33.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.861+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65441d8340 con 0x7f654413f710 2024-03-25T19:42:33.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.866+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912636284 0 0) 0x7f65440abd70 con 0x7f65440a98f0 2024-03-25T19:42:33.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.866+0000 7f653a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.866+0000 7f653a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.866+0000 7f653a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.866+0000 7f653a7fc640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:42:33.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.866+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65441d7710 con 0x7f65440a98f0 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.874+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301427211 0 0) 0x7f6528004eb0 con 0x7f65441430c0 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.874+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65440abd70 con 0x7f65441430c0 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.874+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f653c001b10 con 0x7f65441430c0 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.874+0000 7f653a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.874+0000 7f653a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.874+0000 7f653a7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.874 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.875 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:33.875 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:33.875 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:33.875 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:33.875 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:33.875 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:33.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.874+0000 7f653a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.875+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1828414855 0 0) 0x7f65441d8340 con 0x7f654413f710 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.875+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6528004eb0 con 0x7f654413f710 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.875+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6534001b80 con 0x7f654413f710 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.875+0000 7f653a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.875+0000 7f653a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.875+0000 7f653a7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:33.876 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:33.877 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:33.877 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:33.877 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:33.877 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:33.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3088391320 0 0) 0x7f65441d7710 con 0x7f65440a98f0 2024-03-25T19:42:33.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65441d8340 con 0x7f65440a98f0 2024-03-25T19:42:33.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6540001860 con 0x7f65440a98f0 2024-03-25T19:42:33.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:33.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:33.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:33.885 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 718799018 0 0) 0x7f65440abd70 con 0x7f65441430c0 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 >> v1:172.21.15.193:6789/0 conn(0x7f654413f710 legacy=0x7f65441d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 >> v1:172.21.15.71:6789/0 conn(0x7f65440a98f0 legacy=0x7f6544137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 1 monclient: found mon.c 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65441d84d0 con 0x7f65441430c0 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f653a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:33.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 5 monclient: authenticate success, global_id 14315 2024-03-25T19:42:33.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 10 monclient: _renew_subs 2024-03-25T19:42:33.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:33.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65441d41e0 con 0x7f65441430c0 2024-03-25T19:42:33.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 10 client.14315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:33.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 10 monclient: _renew_subs 2024-03-25T19:42:33.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:33.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65441d5150 con 0x7f65441430c0 2024-03-25T19:42:33.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.884+0000 7f654c1ef640 1 librados: init done 2024-03-25T19:42:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.885+0000 7f654c1ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.885+0000 7f654c1ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.885+0000 7f654c1ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:33.885+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6544074cc0 con 0x7f65441430c0 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.201+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f653c002000 con 0x7f65441430c0 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.201+0000 7f653a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.201+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f653c004d00 con 0x7f65441430c0 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.201+0000 7f653a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.201+0000 7f653a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.201+0000 7f653a7fc640 10 monclient: dump: 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:34.201 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:34.202 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:34.202 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:34.202 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:34.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.201+0000 7f653bfff640 4 set_mon_vals no callback set 2024-03-25T19:42:34.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.202+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f653c004f20 con 0x7f65441430c0 2024-03-25T19:42:34.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.202+0000 7f6549f8b640 10 monclient: get_auth_request con 0x7f652806e0f0 auth_method 0 2024-03-25T19:42:34.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.202+0000 7f653a7fc640 10 client.14315.objecter ms_handle_connect 0x7f652806e0f0 2024-03-25T19:42:34.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.213+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f653c054a30 con 0x7f65441430c0 2024-03-25T19:42:34.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.213+0000 7f653a7fc640 10 client.14315.objecter ms_dispatch 0x7f6544000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:34.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.213+0000 7f653a7fc640 3 client.14315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:34.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.213+0000 7f653a7fc640 3 client.14315.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:34.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.213+0000 7f653a7fc640 20 client.14315.objecter dump_active .. 0 homeless 2024-03-25T19:42:34.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.213+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f653c050540 con 0x7f65441430c0 2024-03-25T19:42:34.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.213+0000 7f653a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:34.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.213+0000 7f653a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:34.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.462+0000 7f654c1ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:34.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.462+0000 7f654c1ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:34.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.462+0000 7f654c1ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:42:34.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.462+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65440627b0 con 0x7f65441430c0 2024-03-25T19:42:34.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.497+0000 7f653a7fc640 1 -- 172.21.15.71:0/588088610 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f653c052470 con 0x7f65441430c0 2024-03-25T19:42:34.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.497+0000 7f653a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:34.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.497+0000 7f653a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:34.497 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:42:34.498 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:34.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.499+0000 7f654c1ef640 10 librados: watch_flush enter 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.499+0000 7f654c1ef640 10 librados: watch_flush exit 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.499+0000 7f654c1ef640 20 client.14315.objecter shutdown clearing up homeless session... 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.499+0000 7f654c1ef640 10 client.14315.objecter successfully canceled tick 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 >> v1:172.21.15.71:6800/3911735624 conn(0x7f652806e0f0 legacy=0x7f65280705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 10 monclient: shutdown 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 >> v1:172.21.15.71:6790/0 conn(0x7f65441430c0 legacy=0x7f65441d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 shutdown_connections 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 >> 172.21.15.71:0/588088610 conn(0x7f65440b7ea0 msgr2=0x7f6544074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 shutdown_connections 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 1 -- 172.21.15.71:0/588088610 wait complete. 2024-03-25T19:42:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:34.500+0000 7f654c1ef640 1 librados: shutdown 2024-03-25T19:42:34.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:36.483 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:19.240 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.490 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:19.247 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.526 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:19.282 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.537 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:19.293 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.537 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:19.256 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.545 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:19.264 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.553 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:19.310 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.571 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:19.290 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.583 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:19.340 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.318 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:41.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.432+0000 7f14c49d6640 10 monclient: get_monmap_and_config 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.432+0000 7f14c49d6640 10 monclient: build_initial_monmap 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.432+0000 7f14c49d6640 10 monclient: monmap: 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:42:41.433212+0000 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:42:41.433212+0000 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:42:41.432 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 1 Processor -- start 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 1 -- start start 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 10 monclient: init 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 10 monclient(hunting): picked mon.noname-a con 0x7f14c00b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 10 monclient(hunting): picked mon.noname-b con 0x7f14c00b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 10 monclient(hunting): picked mon.noname-c con 0x7f14c014be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00af940 con 0x7f14c00b4d60 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00b0a70 con 0x7f14c014be90 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00b0d50 con 0x7f14c00b1730 2024-03-25T19:42:41.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.433+0000 7f14c49d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:41.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.434+0000 7f14c49d6640 10 monclient(hunting): authenticate will time out at 4967.745847s 2024-03-25T19:42:41.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.442+0000 7f14bdd9b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f14c00b1730 0x7f14c00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39434/0 (socket says 172.21.15.71:39434) 2024-03-25T19:42:41.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.442+0000 7f14bdd9b640 1 -- 172.21.15.71:0/2755591194 learned_addr learned my addr 172.21.15.71:0/2755591194 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:41.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.454+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491595603 0 0) 0x7f14c00b0d50 con 0x7f14c00b1730 2024-03-25T19:42:41.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.454+0000 7f14bd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.454+0000 7f14bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.454+0000 7f14bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.454+0000 7f14bd59a640 10 monclient(hunting): my global_id is 14318 2024-03-25T19:42:41.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.454+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a8003520 con 0x7f14c00b1730 2024-03-25T19:42:41.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938619417 0 0) 0x7f14c00b0a70 con 0x7f14c014be90 2024-03-25T19:42:41.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:42:41.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c00b0d50 con 0x7f14c014be90 2024-03-25T19:42:41.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189747209 0 0) 0x7f14c00af940 con 0x7f14c00b4d60 2024-03-25T19:42:41.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:42:41.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.462+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c00b0a70 con 0x7f14c00b4d60 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 872788930 0 0) 0x7f14a8003520 con 0x7f14c00b1730 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 >> v1:172.21.15.193:6789/0 conn(0x7f14c014be90 legacy=0x7f14c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 >> v1:172.21.15.71:6789/0 conn(0x7f14c00b4d60 legacy=0x7f14c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 10 monclient: _finish_hunting 0 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 1 monclient: found mon.noname-b 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c00b1030 con 0x7f14c00b1730 2024-03-25T19:42:41.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 10 monclient: _finish_auth 0 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 10 monclient: _check_auth_tickets 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f14b4001690 con 0x7f14c00b1730 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14c49d6640 5 monclient: authenticate success, global_id 14318 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14c49d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14bd59a640 10 monclient: dump: 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.477 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.478 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:41.478 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:41.478 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:41.478 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:41.478 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:41.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.476+0000 7f14c49d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14b40017f0 con 0x7f14c00b1730 2024-03-25T19:42:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14bd59a640 1 -- 172.21.15.71:0/2755591194 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f14b4001c00 con 0x7f14c00b1730 2024-03-25T19:42:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14c49d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14c49d6640 4 set_mon_vals no callback set 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14bd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14bd59a640 10 monclient: dump: 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:41.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14c49d6640 10 monclient: shutdown 2024-03-25T19:42:41.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14c49d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:41.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.503+0000 7f14c49d6640 1 -- 172.21.15.71:0/2755591194 >> v1:172.21.15.71:6790/0 conn(0x7f14c00b1730 legacy=0x7f14c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.504+0000 7f14c49d6640 1 -- 172.21.15.71:0/2755591194 shutdown_connections 2024-03-25T19:42:41.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.504+0000 7f14c49d6640 1 -- 172.21.15.71:0/2755591194 >> 172.21.15.71:0/2755591194 conn(0x7f14c00617a0 msgr2=0x7f14c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:41.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.504+0000 7f14c49d6640 1 -- 172.21.15.71:0/2755591194 shutdown_connections 2024-03-25T19:42:41.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.504+0000 7f14c49d6640 1 -- 172.21.15.71:0/2755591194 wait complete. 2024-03-25T19:42:41.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.505+0000 7f14c49d6640 10 monclient: build_initial_monmap 2024-03-25T19:42:41.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.505+0000 7f14c49d6640 10 monclient: monmap: 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.505+0000 7f14c49d6640 1 librados: starting msgr at 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.505+0000 7f14c49d6640 1 librados: starting objecter 2024-03-25T19:42:41.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.505+0000 7f14c49d6640 1 Processor -- start 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 1 -- start start 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 1 librados: setting wanted keys 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 1 librados: calling monclient init 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 10 monclient: init 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 10 monclient(hunting): picked mon.noname-a con 0x7f14c00b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 10 monclient(hunting): picked mon.noname-c con 0x7f14c00b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 10 monclient(hunting): picked mon.noname-b con 0x7f14c014be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c013a6d0 con 0x7f14c00b1730 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c0139360 con 0x7f14c014be90 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c0138e40 con 0x7f14c00b4d60 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.506+0000 7f14c49d6640 10 monclient(hunting): authenticate will time out at 4967.818806s 2024-03-25T19:42:41.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.520+0000 7f14bed9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f14c014be90 0x7f14c013b470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:39130/0 (socket says 172.21.15.71:39130) 2024-03-25T19:42:41.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.520+0000 7f14bed9d640 1 -- 172.21.15.71:0/3154558284 learned_addr learned my addr 172.21.15.71:0/3154558284 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:41.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.523+0000 7f14aeffd640 10 client.?.objecter ms_handle_connect 0x7f14c00b4d60 2024-03-25T19:42:41.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.523+0000 7f14aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:41.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.523+0000 7f14aeffd640 10 client.?.objecter ms_handle_connect 0x7f14c014be90 2024-03-25T19:42:41.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.523+0000 7f14aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:41.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.526+0000 7f14aeffd640 10 client.?.objecter ms_handle_connect 0x7f14c00b1730 2024-03-25T19:42:41.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.526+0000 7f14aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:41.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.532+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305748305 0 0) 0x7f14c0139360 con 0x7f14c014be90 2024-03-25T19:42:41.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.532+0000 7f14aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.532+0000 7f14aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.532+0000 7f14aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.532+0000 7f14aeffd640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:42:41.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.532+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a0004eb0 con 0x7f14c014be90 2024-03-25T19:42:41.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.534+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267736030 0 0) 0x7f14c013a6d0 con 0x7f14c00b1730 2024-03-25T19:42:41.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.534+0000 7f14aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.534+0000 7f14aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.534+0000 7f14aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.534+0000 7f14aeffd640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:42:41.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.534+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c0139360 con 0x7f14c00b1730 2024-03-25T19:42:41.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.535+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451932532 0 0) 0x7f14c0138e40 con 0x7f14c00b4d60 2024-03-25T19:42:41.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.535+0000 7f14aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.535+0000 7f14aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.535+0000 7f14aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.535+0000 7f14aeffd640 10 monclient(hunting): my global_id is 14321 2024-03-25T19:42:41.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.535+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c013a6d0 con 0x7f14c00b4d60 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.542+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 96837811 0 0) 0x7f14a0004eb0 con 0x7f14c014be90 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.542+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14c0138e40 con 0x7f14c014be90 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.542+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f14b8001c90 con 0x7f14c014be90 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.542+0000 7f14aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.542+0000 7f14aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.542+0000 7f14aeffd640 10 monclient(hunting): dump: 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.542 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.543 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.543 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:41.543 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:41.543 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:41.543 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:41.543 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:41.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.542+0000 7f14aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:41.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.548+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3349107799 0 0) 0x7f14c013a6d0 con 0x7f14c00b4d60 2024-03-25T19:42:41.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.548+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14a0004eb0 con 0x7f14c00b4d60 2024-03-25T19:42:41.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.548+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f14b4001b00 con 0x7f14c00b4d60 2024-03-25T19:42:41.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.548+0000 7f14aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.548+0000 7f14aeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:41.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.548+0000 7f14aeffd640 10 monclient(hunting): dump: 2024-03-25T19:42:41.548 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:41.549 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:41.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2281827711 0 0) 0x7f14c0138e40 con 0x7f14c014be90 2024-03-25T19:42:41.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 >> v1:172.21.15.71:6790/0 conn(0x7f14c00b4d60 legacy=0x7f14c013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 >> v1:172.21.15.71:6789/0 conn(0x7f14c00b1730 legacy=0x7f14c0131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 1 monclient: found mon.b 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c0138fd0 con 0x7f14c014be90 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 5 monclient: authenticate success, global_id 4318 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 10 monclient: _renew_subs 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c01386b0 con 0x7f14c014be90 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 10 monclient: _renew_subs 2024-03-25T19:42:41.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:41.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c0145650 con 0x7f14c014be90 2024-03-25T19:42:41.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.552+0000 7f14c49d6640 1 librados: init done 2024-03-25T19:42:41.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.553+0000 7f14c49d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:41.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.553+0000 7f14c49d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:41.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.553+0000 7f14c49d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:41.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.553+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c01391c0 con 0x7f14c014be90 2024-03-25T19:42:41.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.771+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14b8003970 con 0x7f14c014be90 2024-03-25T19:42:41.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.771+0000 7f14aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:41.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.771+0000 7f14bcd99640 4 set_mon_vals no callback set 2024-03-25T19:42:41.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.813+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f14b8003b60 con 0x7f14c014be90 2024-03-25T19:42:41.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.813+0000 7f14aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.813+0000 7f14aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:41.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.813+0000 7f14aeffd640 10 monclient: dump: 2024-03-25T19:42:41.812 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:41.812 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:41.812 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.813 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:41.813 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.813 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:41.813 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:41.813 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:41.813 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:41.813 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:41.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.821+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f14b8003d40 con 0x7f14c014be90 2024-03-25T19:42:41.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.822+0000 7f14be59c640 10 monclient: get_auth_request con 0x7f14a006df70 auth_method 0 2024-03-25T19:42:41.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.822+0000 7f14aeffd640 10 client.4318.objecter ms_handle_connect 0x7f14a006df70 2024-03-25T19:42:41.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.823+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f14b80548b0 con 0x7f14c014be90 2024-03-25T19:42:41.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.823+0000 7f14aeffd640 10 client.4318.objecter ms_dispatch 0x7f14c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:41.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.823+0000 7f14aeffd640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:41.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.823+0000 7f14aeffd640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:41.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.823+0000 7f14aeffd640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T19:42:41.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.826+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14b80503c0 con 0x7f14c014be90 2024-03-25T19:42:41.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.826+0000 7f14aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:41.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:41.826+0000 7f14aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:42.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.048+0000 7f14c49d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:42.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.048+0000 7f14c49d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:42.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.048+0000 7f14c49d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:42:42.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.048+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14c0138a50 con 0x7f14c014be90 2024-03-25T19:42:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.074+0000 7f14aeffd640 1 -- 172.21.15.71:0/3154558284 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f14b80522f0 con 0x7f14c014be90 2024-03-25T19:42:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.074+0000 7f14aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.074+0000 7f14aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:42.074 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:42:42.075 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 10 librados: watch_flush enter 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 10 librados: watch_flush exit 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 10 client.4318.objecter successfully canceled tick 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 >> v1:172.21.15.71:6800/3911735624 conn(0x7f14a006df70 legacy=0x7f14a0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 10 monclient: shutdown 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 >> v1:172.21.15.193:6789/0 conn(0x7f14c014be90 legacy=0x7f14c013b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 shutdown_connections 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 >> 172.21.15.71:0/3154558284 conn(0x7f14c00617a0 msgr2=0x7f14c013c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 shutdown_connections 2024-03-25T19:42:42.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 1 -- 172.21.15.71:0/3154558284 wait complete. 2024-03-25T19:42:42.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:42.077+0000 7f14c49d6640 1 librados: shutdown 2024-03-25T19:42:42.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:48.895 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.014+0000 7f7b81954640 10 monclient: get_monmap_and_config 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.014+0000 7f7b81954640 10 monclient: build_initial_monmap 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.014+0000 7f7b81954640 10 monclient: monmap: 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:42:49.015329+0000 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:42:49.015329+0000 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:49.014 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:49.015 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:49.015 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:42:49.015 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:42:49.015 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:49.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.015+0000 7f7b81954640 1 Processor -- start 2024-03-25T19:42:49.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.015+0000 7f7b81954640 1 -- start start 2024-03-25T19:42:49.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.015+0000 7f7b81954640 10 monclient: init 2024-03-25T19:42:49.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:49.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:49.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 10 monclient(hunting): picked mon.noname-b con 0x7f7b7c0a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:49.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 10 monclient(hunting): picked mon.noname-a con 0x7f7b7c13f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:49.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 10 monclient(hunting): picked mon.noname-c con 0x7f7b7c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:49.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c142520 con 0x7f7b7c13f710 2024-03-25T19:42:49.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0b28a0 con 0x7f7b7c1430c0 2024-03-25T19:42:49.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0b2a30 con 0x7f7b7c0a98f0 2024-03-25T19:42:49.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:49.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.016+0000 7f7b81954640 10 monclient(hunting): authenticate will time out at 4975.328197s 2024-03-25T19:42:49.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.028+0000 7f7b7b7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7b7c1430c0 0x7f7b7c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:46294/0 (socket says 172.21.15.71:46294) 2024-03-25T19:42:49.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.028+0000 7f7b7b7fe640 1 -- 172.21.15.71:0/832680113 learned_addr learned my addr 172.21.15.71:0/832680113 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:49.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312922488 0 0) 0x7f7b7c0b28a0 con 0x7f7b7c1430c0 2024-03-25T19:42:49.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:42:49.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b64003520 con 0x7f7b7c1430c0 2024-03-25T19:42:49.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481196563 0 0) 0x7f7b7c0b2a30 con 0x7f7b7c0a98f0 2024-03-25T19:42:49.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 10 monclient(hunting): my global_id is 14324 2024-03-25T19:42:49.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.042+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c0b28a0 con 0x7f7b7c0a98f0 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.052+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263966653 0 0) 0x7f7b7c142520 con 0x7f7b7c13f710 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.052+0000 7f7b79ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.052+0000 7f7b79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.052+0000 7f7b79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.052+0000 7f7b79ffb640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.052+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c0b2a30 con 0x7f7b7c13f710 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546110432 0 0) 0x7f7b64003520 con 0x7f7b7c1430c0 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 >> v1:172.21.15.71:6790/0 conn(0x7f7b7c0a98f0 legacy=0x7f7b7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 >> v1:172.21.15.71:6789/0 conn(0x7f7b7c13f710 legacy=0x7f7b7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 1 monclient: found mon.noname-c 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c0b44c0 con 0x7f7b7c1430c0 2024-03-25T19:42:49.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b74001a10 con 0x7f7b7c1430c0 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b79ffb640 10 monclient: dump: 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:49.058 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:49.059 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:49.059 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:49.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b81954640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:42:49.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.057+0000 7f7b81954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.067+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b74001b70 con 0x7f7b7c1430c0 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.067+0000 7f7b79ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.067+0000 7f7b79ffb640 1 -- 172.21.15.71:0/832680113 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b74001f80 con 0x7f7b7c1430c0 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.067+0000 7f7b79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.067+0000 7f7b79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.067+0000 7f7b79ffb640 10 monclient: dump: 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:49.067 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.067+0000 7f7b81954640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:49.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.067+0000 7f7b81954640 4 set_mon_vals no callback set 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.068+0000 7f7b81954640 10 monclient: shutdown 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.068+0000 7f7b81954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.068+0000 7f7b81954640 1 -- 172.21.15.71:0/832680113 >> v1:172.21.15.193:6789/0 conn(0x7f7b7c1430c0 legacy=0x7f7b7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.068+0000 7f7b81954640 1 -- 172.21.15.71:0/832680113 shutdown_connections 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.068+0000 7f7b81954640 1 -- 172.21.15.71:0/832680113 >> 172.21.15.71:0/832680113 conn(0x7f7b7c0b7ea0 msgr2=0x7f7b7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.068+0000 7f7b81954640 1 -- 172.21.15.71:0/832680113 shutdown_connections 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.068+0000 7f7b81954640 1 -- 172.21.15.71:0/832680113 wait complete. 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.069+0000 7f7b81954640 10 monclient: build_initial_monmap 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.069+0000 7f7b81954640 10 monclient: monmap: 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:49.070 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.069+0000 7f7b81954640 1 librados: starting msgr at 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.069+0000 7f7b81954640 1 librados: starting objecter 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.070+0000 7f7b81954640 1 Processor -- start 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.070+0000 7f7b81954640 1 -- start start 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.070+0000 7f7b81954640 1 librados: setting wanted keys 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.070+0000 7f7b81954640 1 librados: calling monclient init 2024-03-25T19:42:49.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.070+0000 7f7b81954640 10 monclient: init 2024-03-25T19:42:49.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:49.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:49.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 10 monclient(hunting): picked mon.noname-c con 0x7f7b7c0a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:49.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 10 monclient(hunting): picked mon.noname-a con 0x7f7b7c13f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:49.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 10 monclient(hunting): picked mon.noname-b con 0x7f7b7c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:49.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c135230 con 0x7f7b7c13f710 2024-03-25T19:42:49.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c135da0 con 0x7f7b7c1430c0 2024-03-25T19:42:49.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c135f30 con 0x7f7b7c0a98f0 2024-03-25T19:42:49.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:49.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.071+0000 7f7b81954640 10 monclient(hunting): authenticate will time out at 4975.383112s 2024-03-25T19:42:49.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.084+0000 7f7b7affd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7b7c0a98f0 0x7f7b7c1d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50982/0 (socket says 172.21.15.71:50982) 2024-03-25T19:42:49.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.084+0000 7f7b7affd640 1 -- 172.21.15.71:0/1879206629 learned_addr learned my addr 172.21.15.71:0/1879206629 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:49.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.084+0000 7f7b63fff640 10 client.?.objecter ms_handle_connect 0x7f7b7c0a98f0 2024-03-25T19:42:49.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.084+0000 7f7b63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:49.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.089+0000 7f7b63fff640 10 client.?.objecter ms_handle_connect 0x7f7b7c13f710 2024-03-25T19:42:49.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.089+0000 7f7b63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:49.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.094+0000 7f7b63fff640 10 client.?.objecter ms_handle_connect 0x7f7b7c1430c0 2024-03-25T19:42:49.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.094+0000 7f7b63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:49.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.106+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159657752 0 0) 0x7f7b7c135230 con 0x7f7b7c13f710 2024-03-25T19:42:49.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.106+0000 7f7b63fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.106+0000 7f7b63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.106+0000 7f7b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.106+0000 7f7b63fff640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:42:49.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.106+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b68004eb0 con 0x7f7b7c13f710 2024-03-25T19:42:49.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.110+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132107681 0 0) 0x7f7b7c135f30 con 0x7f7b7c0a98f0 2024-03-25T19:42:49.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.110+0000 7f7b63fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.110+0000 7f7b63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.110+0000 7f7b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.110+0000 7f7b63fff640 10 monclient(hunting): my global_id is 14327 2024-03-25T19:42:49.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.110+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c135230 con 0x7f7b7c0a98f0 2024-03-25T19:42:49.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.111+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522205109 0 0) 0x7f7b7c135da0 con 0x7f7b7c1430c0 2024-03-25T19:42:49.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.111+0000 7f7b63fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.111+0000 7f7b63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.111+0000 7f7b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.111+0000 7f7b63fff640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:42:49.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.111+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c135f30 con 0x7f7b7c1430c0 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.119+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1791495043 0 0) 0x7f7b68004eb0 con 0x7f7b7c13f710 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.119+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b7c135da0 con 0x7f7b7c13f710 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.119+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b70001970 con 0x7f7b7c13f710 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.119+0000 7f7b63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.119+0000 7f7b63fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.119+0000 7f7b63fff640 10 monclient(hunting): dump: 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.119 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.120 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:49.120 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:49.120 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:49.120 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:49.120 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:49.120 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:49.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.119+0000 7f7b63fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:49.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.134+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1257977943 0 0) 0x7f7b7c135f30 con 0x7f7b7c1430c0 2024-03-25T19:42:49.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.134+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b68004eb0 con 0x7f7b7c1430c0 2024-03-25T19:42:49.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.134+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b74001e80 con 0x7f7b7c1430c0 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.134+0000 7f7b63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.134+0000 7f7b63fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.134+0000 7f7b63fff640 10 monclient(hunting): dump: 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:49.134 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:49.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3100530628 0 0) 0x7f7b7c135da0 con 0x7f7b7c13f710 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 >> v1:172.21.15.71:6790/0 conn(0x7f7b7c0a98f0 legacy=0x7f7b7c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 >> v1:172.21.15.193:6789/0 conn(0x7f7b7c1430c0 legacy=0x7f7b7c1db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 1 monclient: found mon.a 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c1360c0 con 0x7f7b7c13f710 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:42:49.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 10 monclient: _renew_subs 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c1d5000 con 0x7f7b7c13f710 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 10 monclient: _renew_subs 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c1d41e0 con 0x7f7b7c13f710 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.135+0000 7f7b81954640 1 librados: init done 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.136+0000 7f7b81954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.136+0000 7f7b81954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.136+0000 7f7b81954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:49.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.136+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c135090 con 0x7f7b7c13f710 2024-03-25T19:42:49.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.469+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b70003610 con 0x7f7b7c13f710 2024-03-25T19:42:49.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.469+0000 7f7b63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:49.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.469+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b70003840 con 0x7f7b7c13f710 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.469+0000 7f7b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.469+0000 7f7b63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.469+0000 7f7b63fff640 10 monclient: dump: 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:49.470 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:49.471 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:49.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.469+0000 7f7b797fa640 4 set_mon_vals no callback set 2024-03-25T19:42:49.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.470+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f7b70003a60 con 0x7f7b7c13f710 2024-03-25T19:42:49.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.471+0000 7f7b7affd640 10 monclient: get_auth_request con 0x7f7b6806df70 auth_method 0 2024-03-25T19:42:49.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.471+0000 7f7b63fff640 10 client.4317.objecter ms_handle_connect 0x7f7b6806df70 2024-03-25T19:42:49.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.476+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f7b700534f0 con 0x7f7b7c13f710 2024-03-25T19:42:49.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.476+0000 7f7b63fff640 10 client.4317.objecter ms_dispatch 0x7f7b7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:49.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.476+0000 7f7b63fff640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:49.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.476+0000 7f7b63fff640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:49.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.477+0000 7f7b63fff640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T19:42:49.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.477+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b7004f000 con 0x7f7b7c13f710 2024-03-25T19:42:49.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.477+0000 7f7b63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:49.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.477+0000 7f7b63fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:49.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.731+0000 7f7b81954640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:49.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.731+0000 7f7b81954640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:49.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.731+0000 7f7b81954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:49.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.731+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b7c13b6e0 con 0x7f7b7c13f710 2024-03-25T19:42:49.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.775+0000 7f7b63fff640 1 -- 172.21.15.71:0/1879206629 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f7b70050f30 con 0x7f7b7c13f710 2024-03-25T19:42:49.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.775+0000 7f7b63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:49.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.775+0000 7f7b63fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:49.776 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:42:49.776 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 10 librados: watch_flush enter 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 10 librados: watch_flush exit 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 10 client.4317.objecter successfully canceled tick 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 >> v1:172.21.15.71:6800/3911735624 conn(0x7f7b6806df70 legacy=0x7f7b68070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 10 monclient: shutdown 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 >> v1:172.21.15.71:6789/0 conn(0x7f7b7c13f710 legacy=0x7f7b7c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 shutdown_connections 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.777+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 >> 172.21.15.71:0/1879206629 conn(0x7f7b7c0b7ea0 msgr2=0x7f7b7c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.778+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 shutdown_connections 2024-03-25T19:42:49.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.778+0000 7f7b81954640 1 -- 172.21.15.71:0/1879206629 wait complete. 2024-03-25T19:42:49.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:49.778+0000 7f7b81954640 1 librados: shutdown 2024-03-25T19:42:49.787 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:56.557 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:39.313 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.565 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:39.322 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.597 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:56.598 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:39.354 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.613 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:39.370 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.626 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:39.345 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.628 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:39.347 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.629 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:39.386 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.653 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:39.371 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.660 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:39.417 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.702+0000 7ff7c3cc1640 10 monclient: get_monmap_and_config 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.702+0000 7ff7c3cc1640 10 monclient: build_initial_monmap 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.702+0000 7ff7c3cc1640 10 monclient: monmap: 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:42:56.704440+0000 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:42:56.704440+0000 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:56.703 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:56.704 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:56.704 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:42:56.704 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:42:56.704 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.703+0000 7ff7c3cc1640 1 Processor -- start 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.703+0000 7ff7c3cc1640 1 -- start start 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.703+0000 7ff7c3cc1640 10 monclient: init 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.703+0000 7ff7c3cc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.703+0000 7ff7c3cc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.703+0000 7ff7c3cc1640 10 monclient(hunting): picked mon.noname-a con 0x7ff7bc0a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.703+0000 7ff7c3cc1640 10 monclient(hunting): picked mon.noname-b con 0x7ff7bc13f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:56.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.703+0000 7ff7c3cc1640 10 monclient(hunting): picked mon.noname-c con 0x7ff7bc1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:56.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.704+0000 7ff7c3cc1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc142520 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.704+0000 7ff7c3cc1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc0b28a0 con 0x7ff7bc1430c0 2024-03-25T19:42:56.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.704+0000 7ff7c3cc1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc0b2a30 con 0x7ff7bc13f710 2024-03-25T19:42:56.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.704+0000 7ff7c3cc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:56.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.704+0000 7ff7c3cc1640 10 monclient(hunting): authenticate will time out at 4983.016884s 2024-03-25T19:42:56.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.718+0000 7ff7c125c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7ff7bc13f710 0x7ff7bc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46414/0 (socket says 172.21.15.71:46414) 2024-03-25T19:42:56.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.718+0000 7ff7c125c640 1 -- 172.21.15.71:0/779571250 learned_addr learned my addr 172.21.15.71:0/779571250 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:56.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.766+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508571070 0 0) 0x7ff7bc142520 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.766+0000 7ff7c0a5b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.766+0000 7ff7c0a5b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.766+0000 7ff7c0a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.766+0000 7ff7c0a5b640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:42:56.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.766+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a8003520 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.836+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015114836 0 0) 0x7ff7bc0b28a0 con 0x7ff7bc1430c0 2024-03-25T19:42:56.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.836+0000 7ff7c0a5b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.836+0000 7ff7c0a5b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.836+0000 7ff7c0a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.836+0000 7ff7c0a5b640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:42:56.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.836+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7bc142520 con 0x7ff7bc1430c0 2024-03-25T19:42:56.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.845+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734614268 0 0) 0x7ff7bc0b2a30 con 0x7ff7bc13f710 2024-03-25T19:42:56.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.845+0000 7ff7c0a5b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.845+0000 7ff7c0a5b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.845+0000 7ff7c0a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.845+0000 7ff7c0a5b640 10 monclient(hunting): my global_id is 14330 2024-03-25T19:42:56.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.845+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7bc0b28a0 con 0x7ff7bc13f710 2024-03-25T19:42:56.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4023324109 0 0) 0x7ff7a8003520 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 >> v1:172.21.15.71:6790/0 conn(0x7ff7bc13f710 legacy=0x7ff7bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 >> v1:172.21.15.193:6789/0 conn(0x7ff7bc1430c0 legacy=0x7ff7bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:56.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 1 monclient: found mon.noname-a 2024-03-25T19:42:56.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:56.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc0b44c0 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 10 monclient: _finish_auth 0 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff7b8001690 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c0a5b640 10 monclient: dump: 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:56.855 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:56.856 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:56.856 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:56.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c3cc1640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:42:56.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.853+0000 7ff7c3cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:56.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.871+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7b80017f0 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.871+0000 7ff7c0a5b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:56.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.871+0000 7ff7c0a5b640 1 -- 172.21.15.71:0/779571250 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff7b8001c00 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.871+0000 7ff7c0a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:56.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.871+0000 7ff7c0a5b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:56.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.871+0000 7ff7c0a5b640 10 monclient: dump: 2024-03-25T19:42:56.872 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:56.872 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.871+0000 7ff7c3cc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:56.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.871+0000 7ff7c3cc1640 4 set_mon_vals no callback set 2024-03-25T19:42:56.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.872+0000 7ff7c3cc1640 10 monclient: shutdown 2024-03-25T19:42:56.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.872+0000 7ff7c3cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:56.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.872+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/779571250 >> v1:172.21.15.71:6789/0 conn(0x7ff7bc0a98f0 legacy=0x7ff7bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.872+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/779571250 shutdown_connections 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.872+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/779571250 >> 172.21.15.71:0/779571250 conn(0x7ff7bc0b7ea0 msgr2=0x7ff7bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.872+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/779571250 shutdown_connections 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.872+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/779571250 wait complete. 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 10 monclient: build_initial_monmap 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 10 monclient: monmap: 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:42:56.876 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 1 librados: starting msgr at 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 1 librados: starting objecter 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 1 Processor -- start 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 1 -- start start 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 1 librados: setting wanted keys 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 1 librados: calling monclient init 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.873+0000 7ff7c3cc1640 10 monclient: init 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 10 monclient(hunting): picked mon.noname-a con 0x7ff7bc0a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 10 monclient(hunting): picked mon.noname-b con 0x7ff7bc13f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:42:56.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 10 monclient(hunting): picked mon.noname-c con 0x7ff7bc1430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:42:56.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc138970 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc1d7810 con 0x7ff7bc13f710 2024-03-25T19:42:56.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc139990 con 0x7ff7bc1430c0 2024-03-25T19:42:56.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:56.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.874+0000 7ff7c3cc1640 10 monclient(hunting): authenticate will time out at 4983.187241s 2024-03-25T19:42:56.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.881+0000 7ff7c1a5d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff7bc0a98f0 0x7ff7bc138260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37348/0 (socket says 172.21.15.71:37348) 2024-03-25T19:42:56.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.881+0000 7ff7c1a5d640 1 -- 172.21.15.71:0/517538285 learned_addr learned my addr 172.21.15.71:0/517538285 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:42:56.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.886+0000 7ff7b27fc640 10 client.?.objecter ms_handle_connect 0x7ff7bc0a98f0 2024-03-25T19:42:56.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.886+0000 7ff7b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:56.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.895+0000 7ff7b27fc640 10 client.?.objecter ms_handle_connect 0x7ff7bc1430c0 2024-03-25T19:42:56.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.895+0000 7ff7b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:56.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.900+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331524833 0 0) 0x7ff7bc139990 con 0x7ff7bc1430c0 2024-03-25T19:42:56.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.901+0000 7ff7b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.901+0000 7ff7b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.901+0000 7ff7b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.901+0000 7ff7b27fc640 10 monclient(hunting): my global_id is 14333 2024-03-25T19:42:56.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.901+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a0004eb0 con 0x7ff7bc1430c0 2024-03-25T19:42:56.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.906+0000 7ff7b27fc640 10 client.?.objecter ms_handle_connect 0x7ff7bc13f710 2024-03-25T19:42:56.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.906+0000 7ff7b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:56.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.910+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828725260 0 0) 0x7ff7bc138970 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.910+0000 7ff7b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.910+0000 7ff7b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.910+0000 7ff7b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.910+0000 7ff7b27fc640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:42:56.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.910+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7bc139990 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.914+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399360140 0 0) 0x7ff7a0004eb0 con 0x7ff7bc1430c0 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.914+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7bc138970 con 0x7ff7bc1430c0 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.914+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff7b4001bd0 con 0x7ff7bc1430c0 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.914+0000 7ff7b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.914+0000 7ff7b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.914+0000 7ff7b27fc640 10 monclient(hunting): dump: 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:56.915 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:56.916 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:56.916 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:56.916 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:56.916 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:56.916 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:56.916 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:56.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.914+0000 7ff7b27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:56.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.918+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590841538 0 0) 0x7ff7bc1d7810 con 0x7ff7bc13f710 2024-03-25T19:42:56.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.918+0000 7ff7b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.918+0000 7ff7b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.918+0000 7ff7b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.918+0000 7ff7b27fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:42:56.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.918+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a0004eb0 con 0x7ff7bc13f710 2024-03-25T19:42:56.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.920+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 264999842 0 0) 0x7ff7bc139990 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.920+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7bc1d7810 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.920+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff7b8001b40 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.920+0000 7ff7b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.920+0000 7ff7b27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.920+0000 7ff7b27fc640 10 monclient(hunting): dump: 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:56.922 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:56.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3611176557 0 0) 0x7ff7bc1d7810 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 >> v1:172.21.15.71:6790/0 conn(0x7ff7bc1430c0 legacy=0x7ff7bc1d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 >> v1:172.21.15.193:6789/0 conn(0x7ff7bc13f710 legacy=0x7ff7bc1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:56.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 1 monclient: found mon.a 2024-03-25T19:42:56.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc1d8440 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 5 monclient: authenticate success, global_id 4323 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 10 monclient: _renew_subs 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7bc1d41e0 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 10 monclient: _renew_subs 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:56.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc138d80 con 0x7ff7bc0a98f0 2024-03-25T19:42:56.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.929+0000 7ff7c3cc1640 1 librados: init done 2024-03-25T19:42:56.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.930+0000 7ff7c3cc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:56.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.930+0000 7ff7c3cc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:56.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.930+0000 7ff7c3cc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:56.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:56.930+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7bc1d79a0 con 0x7ff7bc0a98f0 2024-03-25T19:42:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.241+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7b80011f0 con 0x7ff7bc0a98f0 2024-03-25T19:42:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.241+0000 7ff7b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.241+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff7b8003a40 con 0x7ff7bc0a98f0 2024-03-25T19:42:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.241+0000 7ff7b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.241+0000 7ff7b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.241+0000 7ff7b27fc640 10 monclient: dump: 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:42:57.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.241+0000 7ff7b3fff640 4 set_mon_vals no callback set 2024-03-25T19:42:57.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.241+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7ff7b8003c60 con 0x7ff7bc0a98f0 2024-03-25T19:42:57.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.242+0000 7ff7c125c640 10 monclient: get_auth_request con 0x7ff7a006df00 auth_method 0 2024-03-25T19:42:57.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.242+0000 7ff7b27fc640 10 client.4323.objecter ms_handle_connect 0x7ff7a006df00 2024-03-25T19:42:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.248+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7ff7b8054860 con 0x7ff7bc0a98f0 2024-03-25T19:42:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.248+0000 7ff7b27fc640 10 client.4323.objecter ms_dispatch 0x7ff7bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.248+0000 7ff7b27fc640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.248+0000 7ff7b27fc640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.248+0000 7ff7b27fc640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T19:42:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.248+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7b80501a0 con 0x7ff7bc0a98f0 2024-03-25T19:42:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.248+0000 7ff7b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.248+0000 7ff7b27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:57.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.502+0000 7ff7c3cc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:57.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.502+0000 7ff7c3cc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:57.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.502+0000 7ff7c3cc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:42:57.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.502+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7bc0627b0 con 0x7ff7bc0a98f0 2024-03-25T19:42:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.527+0000 7ff7b27fc640 1 -- 172.21.15.71:0/517538285 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7ff7b80522a0 con 0x7ff7bc0a98f0 2024-03-25T19:42:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.527+0000 7ff7b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.527+0000 7ff7b27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:57.528 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:42:57.528 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 10 librados: watch_flush enter 2024-03-25T19:42:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 10 librados: watch_flush exit 2024-03-25T19:42:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T19:42:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 10 client.4323.objecter successfully canceled tick 2024-03-25T19:42:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 >> v1:172.21.15.71:6800/3911735624 conn(0x7ff7a006df00 legacy=0x7ff7a00703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 10 monclient: shutdown 2024-03-25T19:42:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 >> v1:172.21.15.71:6789/0 conn(0x7ff7bc0a98f0 legacy=0x7ff7bc138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 shutdown_connections 2024-03-25T19:42:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 >> 172.21.15.71:0/517538285 conn(0x7ff7bc0b7ea0 msgr2=0x7ff7bc13c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 shutdown_connections 2024-03-25T19:42:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.530+0000 7ff7c3cc1640 1 -- 172.21.15.71:0/517538285 wait complete. 2024-03-25T19:42:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:42:57.531+0000 7ff7c3cc1640 1 librados: shutdown 2024-03-25T19:42:57.543 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:04.349 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:04.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.468+0000 7f24d9f04640 10 monclient: get_monmap_and_config 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.468+0000 7f24d9f04640 10 monclient: build_initial_monmap 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.468+0000 7f24d9f04640 10 monclient: monmap: 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:43:04.469176+0000 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:43:04.469176+0000 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:04.468 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:04.469 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:43:04.469 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:43:04.469 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:04.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.469+0000 7f24d9f04640 1 Processor -- start 2024-03-25T19:43:04.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.469+0000 7f24d9f04640 1 -- start start 2024-03-25T19:43:04.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.469+0000 7f24d9f04640 10 monclient: init 2024-03-25T19:43:04.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.469+0000 7f24d9f04640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:04.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.469+0000 7f24d9f04640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:04.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.470+0000 7f24d9f04640 10 monclient(hunting): picked mon.noname-b con 0x7f24d40a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:04.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.470+0000 7f24d9f04640 10 monclient(hunting): picked mon.noname-c con 0x7f24d413f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:04.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.470+0000 7f24d9f04640 10 monclient(hunting): picked mon.noname-a con 0x7f24d41430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:04.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.470+0000 7f24d9f04640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d4142520 con 0x7f24d41430c0 2024-03-25T19:43:04.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.470+0000 7f24d9f04640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d40b28a0 con 0x7f24d413f710 2024-03-25T19:43:04.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.470+0000 7f24d9f04640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d40b2a30 con 0x7f24d40a98f0 2024-03-25T19:43:04.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.470+0000 7f24d9f04640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:04.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.470+0000 7f24d9f04640 10 monclient(hunting): authenticate will time out at 4990.782100s 2024-03-25T19:43:04.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.477+0000 7f24d3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f24d41430c0 0x7f24d414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41982/0 (socket says 172.21.15.71:41982) 2024-03-25T19:43:04.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.477+0000 7f24d3fff640 1 -- 172.21.15.71:0/2518311753 learned_addr learned my addr 172.21.15.71:0/2518311753 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:04.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.488+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022649131 0 0) 0x7f24d4142520 con 0x7f24d41430c0 2024-03-25T19:43:04.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.488+0000 7f24d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:04.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.488+0000 7f24d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:04.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.488+0000 7f24d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:04.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.488+0000 7f24d27fc640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:43:04.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.489+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24bc003520 con 0x7f24d41430c0 2024-03-25T19:43:04.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938341519 0 0) 0x7f24d40b28a0 con 0x7f24d413f710 2024-03-25T19:43:04.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:04.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:04.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:04.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:43:04.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d4142520 con 0x7f24d413f710 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 222437580 0 0) 0x7f24bc003520 con 0x7f24d41430c0 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 >> v1:172.21.15.71:6790/0 conn(0x7f24d40a98f0 legacy=0x7f24d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 >> v1:172.21.15.193:6789/0 conn(0x7f24d413f710 legacy=0x7f24d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 1 monclient: found mon.noname-a 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d40b44c0 con 0x7f24d41430c0 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:04.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f24cc001a10 con 0x7f24d41430c0 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d9f04640 5 monclient: authenticate success, global_id 4326 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d9f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d27fc640 10 monclient: dump: 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:04.509 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:04.510 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:04.510 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:04.510 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:04.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.507+0000 7f24d9f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:04.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.516+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24cc001b70 con 0x7f24d41430c0 2024-03-25T19:43:04.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.516+0000 7f24d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:04.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.516+0000 7f24d27fc640 1 -- 172.21.15.71:0/2518311753 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f24cc001f80 con 0x7f24d41430c0 2024-03-25T19:43:04.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.516+0000 7f24d9f04640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:04.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.516+0000 7f24d9f04640 4 set_mon_vals no callback set 2024-03-25T19:43:04.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.516+0000 7f24d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.516+0000 7f24d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.516+0000 7f24d27fc640 10 monclient: dump: 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:04.517 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.517+0000 7f24d9f04640 10 monclient: shutdown 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.517+0000 7f24d9f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.517+0000 7f24d9f04640 1 -- 172.21.15.71:0/2518311753 >> v1:172.21.15.71:6789/0 conn(0x7f24d41430c0 legacy=0x7f24d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.517+0000 7f24d9f04640 1 -- 172.21.15.71:0/2518311753 shutdown_connections 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.517+0000 7f24d9f04640 1 -- 172.21.15.71:0/2518311753 >> 172.21.15.71:0/2518311753 conn(0x7f24d40b7ea0 msgr2=0x7f24d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.517+0000 7f24d9f04640 1 -- 172.21.15.71:0/2518311753 shutdown_connections 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.517+0000 7f24d9f04640 1 -- 172.21.15.71:0/2518311753 wait complete. 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.518+0000 7f24d9f04640 10 monclient: build_initial_monmap 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.518+0000 7f24d9f04640 10 monclient: monmap: 2024-03-25T19:43:04.519 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:04.520 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:04.520 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.519+0000 7f24d9f04640 1 librados: starting msgr at 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.519+0000 7f24d9f04640 1 librados: starting objecter 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.519+0000 7f24d9f04640 1 Processor -- start 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.519+0000 7f24d9f04640 1 -- start start 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.519+0000 7f24d9f04640 1 librados: setting wanted keys 2024-03-25T19:43:04.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.519+0000 7f24d9f04640 1 librados: calling monclient init 2024-03-25T19:43:04.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.519+0000 7f24d9f04640 10 monclient: init 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 10 monclient(hunting): picked mon.noname-b con 0x7f24d40a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 10 monclient(hunting): picked mon.noname-a con 0x7f24d413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 10 monclient(hunting): picked mon.noname-c con 0x7f24d41430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d4136c50 con 0x7f24d413f710 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d4136ec0 con 0x7f24d40a98f0 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d41317f0 con 0x7f24d41430c0 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:04.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.520+0000 7f24d9f04640 10 monclient(hunting): authenticate will time out at 4990.832219s 2024-03-25T19:43:04.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.532+0000 7f24d2ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f24d413f710 0x7f24d41dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42022/0 (socket says 172.21.15.71:42022) 2024-03-25T19:43:04.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.532+0000 7f24d2ffd640 1 -- 172.21.15.71:0/528899397 learned_addr learned my addr 172.21.15.71:0/528899397 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:04.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.535+0000 7f24bbfff640 10 client.?.objecter ms_handle_connect 0x7f24d413f710 2024-03-25T19:43:04.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.535+0000 7f24bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:04.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.541+0000 7f24bbfff640 10 client.?.objecter ms_handle_connect 0x7f24d40a98f0 2024-03-25T19:43:04.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.541+0000 7f24bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:04.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.546+0000 7f24bbfff640 10 client.?.objecter ms_handle_connect 0x7f24d41430c0 2024-03-25T19:43:04.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.546+0000 7f24bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:04.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239796351 0 0) 0x7f24d4136c50 con 0x7f24d413f710 2024-03-25T19:43:04.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:04.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:04.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:04.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:43:04.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c0004eb0 con 0x7f24d413f710 2024-03-25T19:43:04.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109719734 0 0) 0x7f24d4136ec0 con 0x7f24d40a98f0 2024-03-25T19:43:04.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:04.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:04.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:04.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:43:04.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.552+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d4136c50 con 0x7f24d40a98f0 2024-03-25T19:43:04.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.568+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3734678191 0 0) 0x7f24c0004eb0 con 0x7f24d413f710 2024-03-25T19:43:04.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.568+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d4136ec0 con 0x7f24d413f710 2024-03-25T19:43:04.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.568+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f24c8001a90 con 0x7f24d413f710 2024-03-25T19:43:04.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.568+0000 7f24bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.568+0000 7f24bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:04.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.568+0000 7f24bbfff640 10 monclient(hunting): dump: 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:04.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.568+0000 7f24bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:04.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.570+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2720857794 0 0) 0x7f24d4136c50 con 0x7f24d40a98f0 2024-03-25T19:43:04.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.570+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24c0004eb0 con 0x7f24d40a98f0 2024-03-25T19:43:04.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.570+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f24c40018a0 con 0x7f24d40a98f0 2024-03-25T19:43:04.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.570+0000 7f24bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.570+0000 7f24bbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:04.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.570+0000 7f24bbfff640 10 monclient(hunting): dump: 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:04.571 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:04.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.573+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005618487 0 0) 0x7f24d41317f0 con 0x7f24d41430c0 2024-03-25T19:43:04.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.573+0000 7f24bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:04.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.573+0000 7f24bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:04.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.573+0000 7f24bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:04.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.573+0000 7f24bbfff640 10 monclient(hunting): my global_id is 14339 2024-03-25T19:43:04.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.573+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d4136c50 con 0x7f24d41430c0 2024-03-25T19:43:04.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.584+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 630342612 0 0) 0x7f24d4136c50 con 0x7f24d41430c0 2024-03-25T19:43:04.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.584+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d41317f0 con 0x7f24d41430c0 2024-03-25T19:43:04.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.584+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f24cc001e80 con 0x7f24d41430c0 2024-03-25T19:43:04.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.584+0000 7f24bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.584+0000 7f24bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:04.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.584+0000 7f24bbfff640 10 monclient(hunting): dump: 2024-03-25T19:43:04.584 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:04.585 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3758661978 0 0) 0x7f24d4136ec0 con 0x7f24d413f710 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 >> v1:172.21.15.71:6790/0 conn(0x7f24d41430c0 legacy=0x7f24d41db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 >> v1:172.21.15.193:6789/0 conn(0x7f24d40a98f0 legacy=0x7f24d41310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 1 monclient: found mon.a 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d4131980 con 0x7f24d413f710 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 10 monclient: _finish_auth 0 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:04.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 5 monclient: authenticate success, global_id 4329 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 10 monclient: _renew_subs 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d4135070 con 0x7f24d413f710 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 10 monclient: _renew_subs 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d40b7870 con 0x7f24d413f710 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.585+0000 7f24d9f04640 1 librados: init done 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.586+0000 7f24d9f04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.586+0000 7f24d9f04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.586+0000 7f24d9f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:04.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.586+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d40627b0 con 0x7f24d413f710 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.901+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24c8001cd0 con 0x7f24d413f710 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.901+0000 7f24bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.901+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f24c8003a60 con 0x7f24d413f710 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.901+0000 7f24bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.901+0000 7f24bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.901+0000 7f24bbfff640 10 monclient: dump: 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:04.901 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.901+0000 7f24d1ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:04.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.901+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f24c8003c80 con 0x7f24d413f710 2024-03-25T19:43:04.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.902+0000 7f24d37fe640 10 monclient: get_auth_request con 0x7f24c006e130 auth_method 0 2024-03-25T19:43:04.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.902+0000 7f24bbfff640 10 client.4329.objecter ms_handle_connect 0x7f24c006e130 2024-03-25T19:43:04.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.907+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f24c80548c0 con 0x7f24d413f710 2024-03-25T19:43:04.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.907+0000 7f24bbfff640 10 client.4329.objecter ms_dispatch 0x7f24d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:04.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.907+0000 7f24bbfff640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:04.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.907+0000 7f24bbfff640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:04.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.907+0000 7f24bbfff640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-25T19:43:04.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.907+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24c80503d0 con 0x7f24d413f710 2024-03-25T19:43:04.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.907+0000 7f24bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:04.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:04.907+0000 7f24bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:05.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.121+0000 7f24d9f04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:05.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.121+0000 7f24d9f04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:05.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.121+0000 7f24d9f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:05.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.121+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24d40b7ac0 con 0x7f24d413f710 2024-03-25T19:43:05.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.245+0000 7f24bbfff640 1 -- 172.21.15.71:0/528899397 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f24c8052300 con 0x7f24d413f710 2024-03-25T19:43:05.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.246+0000 7f24bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:05.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.246+0000 7f24bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:05.246 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:43:05.246 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:05.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 10 librados: watch_flush enter 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 10 librados: watch_flush exit 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 10 client.4329.objecter successfully canceled tick 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 >> v1:172.21.15.71:6800/3911735624 conn(0x7f24c006e130 legacy=0x7f24c00705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 10 monclient: shutdown 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 >> v1:172.21.15.71:6789/0 conn(0x7f24d413f710 legacy=0x7f24d41dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 shutdown_connections 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 >> 172.21.15.71:0/528899397 conn(0x7f24d40b7ea0 msgr2=0x7f24d413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.248+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 shutdown_connections 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.249+0000 7f24d9f04640 1 -- 172.21.15.71:0/528899397 wait complete. 2024-03-25T19:43:05.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:05.249+0000 7f24d9f04640 1 librados: shutdown 2024-03-25T19:43:05.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:12.067 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.181+0000 7f9a951cf640 10 monclient: get_monmap_and_config 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.181+0000 7f9a951cf640 10 monclient: build_initial_monmap 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.181+0000 7f9a951cf640 10 monclient: monmap: 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:43:12.182574+0000 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:43:12.182574+0000 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:12.181 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:43:12.182 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:43:12.182 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:12.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.182+0000 7f9a951cf640 1 Processor -- start 2024-03-25T19:43:12.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.182+0000 7f9a951cf640 1 -- start start 2024-03-25T19:43:12.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.182+0000 7f9a951cf640 10 monclient: init 2024-03-25T19:43:12.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:12.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:12.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 10 monclient(hunting): picked mon.noname-c con 0x7f9a9013fc60 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:12.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 10 monclient(hunting): picked mon.noname-a con 0x7f9a90070570 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:12.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 10 monclient(hunting): picked mon.noname-b con 0x7f9a900727a0 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:12.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9013c9f0 con 0x7f9a90070570 2024-03-25T19:43:12.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9006fac0 con 0x7f9a9013fc60 2024-03-25T19:43:12.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a900715d0 con 0x7f9a900727a0 2024-03-25T19:43:12.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:12.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.183+0000 7f9a951cf640 10 monclient(hunting): authenticate will time out at 4998.495141s 2024-03-25T19:43:12.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.187+0000 7f9a8e59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9a90070570 0x7f9a90072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54876/0 (socket says 172.21.15.71:54876) 2024-03-25T19:43:12.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.187+0000 7f9a8e59c640 1 -- 172.21.15.71:0/4145621063 learned_addr learned my addr 172.21.15.71:0/4145621063 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:12.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.201+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471812173 0 0) 0x7f9a9013c9f0 con 0x7f9a90070570 2024-03-25T19:43:12.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.201+0000 7f9a8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:12.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.201+0000 7f9a8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:12.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.201+0000 7f9a8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:12.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.201+0000 7f9a8dd9b640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:43:12.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.201+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a78003520 con 0x7f9a90070570 2024-03-25T19:43:12.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.208+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358035069 0 0) 0x7f9a900715d0 con 0x7f9a900727a0 2024-03-25T19:43:12.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.209+0000 7f9a8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:12.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.209+0000 7f9a8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:12.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.209+0000 7f9a8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:12.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.209+0000 7f9a8dd9b640 10 monclient(hunting): my global_id is 14342 2024-03-25T19:43:12.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.209+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a9013c9f0 con 0x7f9a900727a0 2024-03-25T19:43:12.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.210+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653501174 0 0) 0x7f9a9006fac0 con 0x7f9a9013fc60 2024-03-25T19:43:12.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.210+0000 7f9a8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:12.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.210+0000 7f9a8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:12.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.210+0000 7f9a8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:12.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.210+0000 7f9a8dd9b640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:43:12.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.210+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a900715d0 con 0x7f9a9013fc60 2024-03-25T19:43:12.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4063423614 0 0) 0x7f9a78003520 con 0x7f9a90070570 2024-03-25T19:43:12.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 >> v1:172.21.15.71:6790/0 conn(0x7f9a900727a0 legacy=0x7f9a9014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 >> v1:172.21.15.193:6789/0 conn(0x7f9a9013fc60 legacy=0x7f9a9006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a900718b0 con 0x7f9a90070570 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a951cf640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a951cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a84001690 con 0x7f9a90070570 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a8dd9b640 10 monclient: dump: 2024-03-25T19:43:12.213 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:12.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.212+0000 7f9a951cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:12.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.228+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a840017f0 con 0x7f9a90070570 2024-03-25T19:43:12.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.228+0000 7f9a8dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:12.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.228+0000 7f9a8dd9b640 1 -- 172.21.15.71:0/4145621063 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a84001c00 con 0x7f9a90070570 2024-03-25T19:43:12.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.228+0000 7f9a8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:12.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a8dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a8dd9b640 10 monclient: dump: 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a951cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:12.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a951cf640 4 set_mon_vals no callback set 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a951cf640 10 monclient: shutdown 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a951cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a951cf640 1 -- 172.21.15.71:0/4145621063 >> v1:172.21.15.71:6789/0 conn(0x7f9a90070570 legacy=0x7f9a90072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a951cf640 1 -- 172.21.15.71:0/4145621063 shutdown_connections 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.229+0000 7f9a951cf640 1 -- 172.21.15.71:0/4145621063 >> 172.21.15.71:0/4145621063 conn(0x7f9a900b4a80 msgr2=0x7f9a900b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.230+0000 7f9a951cf640 1 -- 172.21.15.71:0/4145621063 shutdown_connections 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.230+0000 7f9a951cf640 1 -- 172.21.15.71:0/4145621063 wait complete. 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.230+0000 7f9a951cf640 10 monclient: build_initial_monmap 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.230+0000 7f9a951cf640 10 monclient: monmap: 2024-03-25T19:43:12.232 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.231+0000 7f9a951cf640 1 librados: starting msgr at 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.231+0000 7f9a951cf640 1 librados: starting objecter 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.231+0000 7f9a951cf640 1 Processor -- start 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.231+0000 7f9a951cf640 1 -- start start 2024-03-25T19:43:12.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.231+0000 7f9a951cf640 1 librados: setting wanted keys 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.231+0000 7f9a951cf640 1 librados: calling monclient init 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.231+0000 7f9a951cf640 10 monclient: init 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 10 monclient(hunting): picked mon.noname-b con 0x7f9a90070570 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 10 monclient(hunting): picked mon.noname-a con 0x7f9a900727a0 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 10 monclient(hunting): picked mon.noname-c con 0x7f9a9013fc60 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a901335e0 con 0x7f9a900727a0 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a901412d0 con 0x7f9a90070570 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a901415b0 con 0x7f9a9013fc60 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:12.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.232+0000 7f9a951cf640 10 monclient(hunting): authenticate will time out at 4998.544433s 2024-03-25T19:43:12.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.237+0000 7f9a8ed9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f9a90070570 0x7f9a901d6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:53708/0 (socket says 172.21.15.71:53708) 2024-03-25T19:43:12.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.237+0000 7f9a8ed9d640 1 -- 172.21.15.71:0/2113574568 learned_addr learned my addr 172.21.15.71:0/2113574568 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:12.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.239+0000 7f9a777fe640 10 client.?.objecter ms_handle_connect 0x7f9a90070570 2024-03-25T19:43:12.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.239+0000 7f9a777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:12.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.243+0000 7f9a777fe640 10 client.?.objecter ms_handle_connect 0x7f9a900727a0 2024-03-25T19:43:12.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.243+0000 7f9a777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:12.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.246+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576921641 0 0) 0x7f9a901412d0 con 0x7f9a90070570 2024-03-25T19:43:12.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.246+0000 7f9a777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:12.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.246+0000 7f9a777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:12.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.246+0000 7f9a777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:12.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.246+0000 7f9a777fe640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:43:12.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.246+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c004eb0 con 0x7f9a90070570 2024-03-25T19:43:12.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.253+0000 7f9a777fe640 10 client.?.objecter ms_handle_connect 0x7f9a9013fc60 2024-03-25T19:43:12.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.253+0000 7f9a777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.258+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2323614517 0 0) 0x7f9a7c004eb0 con 0x7f9a90070570 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.258+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a901412d0 con 0x7f9a90070570 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.258+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a80001970 con 0x7f9a90070570 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.258+0000 7f9a777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.258+0000 7f9a777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.258+0000 7f9a777fe640 10 monclient(hunting): dump: 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:12.258 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:12.259 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:12.259 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:12.259 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:12.259 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:12.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.258+0000 7f9a777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:12.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.262+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1865395695 0 0) 0x7f9a901412d0 con 0x7f9a90070570 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.262+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 >> v1:172.21.15.71:6790/0 conn(0x7f9a9013fc60 legacy=0x7f9a90132ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.262+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 >> v1:172.21.15.71:6789/0 conn(0x7f9a900727a0 legacy=0x7f9a901327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.262+0000 7f9a777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.262+0000 7f9a777fe640 1 monclient: found mon.b 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a90140350 con 0x7f9a90070570 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a777fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 5 monclient: authenticate success, global_id 4342 2024-03-25T19:43:12.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 10 monclient: _renew_subs 2024-03-25T19:43:12.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:12.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a901cb030 con 0x7f9a90070570 2024-03-25T19:43:12.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:12.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 10 monclient: _renew_subs 2024-03-25T19:43:12.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:12.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a90133a40 con 0x7f9a90070570 2024-03-25T19:43:12.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.263+0000 7f9a951cf640 1 librados: init done 2024-03-25T19:43:12.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.264+0000 7f9a951cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:12.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.264+0000 7f9a951cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:12.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.264+0000 7f9a951cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:12.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.264+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a901cae80 con 0x7f9a90070570 2024-03-25T19:43:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.535+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a800035f0 con 0x7f9a90070570 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.535+0000 7f9a777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.535+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9a80003a40 con 0x7f9a90070570 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.535+0000 7f9a777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.535+0000 7f9a777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.535+0000 7f9a777fe640 10 monclient: dump: 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:12.536 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:12.537 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:12.537 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:12.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.535+0000 7f9a8d59a640 4 set_mon_vals no callback set 2024-03-25T19:43:12.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.537+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f9a80003c60 con 0x7f9a90070570 2024-03-25T19:43:12.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.537+0000 7f9a8e59c640 10 monclient: get_auth_request con 0x7f9a7c06dd00 auth_method 0 2024-03-25T19:43:12.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.537+0000 7f9a777fe640 10 client.4342.objecter ms_handle_connect 0x7f9a7c06dd00 2024-03-25T19:43:12.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.543+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f9a800547f0 con 0x7f9a90070570 2024-03-25T19:43:12.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.543+0000 7f9a777fe640 10 client.4342.objecter ms_dispatch 0x7f9a90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:12.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.543+0000 7f9a777fe640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:12.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.543+0000 7f9a777fe640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:12.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.543+0000 7f9a777fe640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T19:43:12.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.544+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a80050300 con 0x7f9a90070570 2024-03-25T19:43:12.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.544+0000 7f9a777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:12.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.544+0000 7f9a777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:12.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.794+0000 7f9a951cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:12.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.794+0000 7f9a951cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:12.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.794+0000 7f9a951cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:12.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.794+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a901404e0 con 0x7f9a90070570 2024-03-25T19:43:12.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.938+0000 7f9a777fe640 1 -- 172.21.15.71:0/2113574568 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f9a80052230 con 0x7f9a90070570 2024-03-25T19:43:12.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.938+0000 7f9a777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:12.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.938+0000 7f9a777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:12.939 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:43:12.939 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 10 librados: watch_flush enter 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 10 librados: watch_flush exit 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 10 client.4342.objecter successfully canceled tick 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 >> v1:172.21.15.71:6800/3911735624 conn(0x7f9a7c06dd00 legacy=0x7f9a7c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 10 monclient: shutdown 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 >> v1:172.21.15.193:6789/0 conn(0x7f9a90070570 legacy=0x7f9a901d6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 shutdown_connections 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 >> 172.21.15.71:0/2113574568 conn(0x7f9a900b4a80 msgr2=0x7f9a90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:12.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 shutdown_connections 2024-03-25T19:43:12.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 1 -- 172.21.15.71:0/2113574568 wait complete. 2024-03-25T19:43:12.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:12.941+0000 7f9a951cf640 1 librados: shutdown 2024-03-25T19:43:12.951 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:16.577 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:59.334 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.625 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:59.381 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.665 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:02:59.421 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.680 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:59.437 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.689 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:59.445 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.693 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:59.412 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.701 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:59.420 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.717 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:02:59.474 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.718 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:59.438 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.758 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.876+0000 7f7d04b15640 10 monclient: get_monmap_and_config 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.876+0000 7f7d04b15640 10 monclient: build_initial_monmap 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.876+0000 7f7d04b15640 10 monclient: monmap: 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:43:19.877470+0000 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:43:19.877470+0000 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:19.876 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:19.877 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:19.877 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:43:19.877 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:43:19.877 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:19.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.877+0000 7f7d04b15640 1 Processor -- start 2024-03-25T19:43:19.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.877+0000 7f7d04b15640 1 -- start start 2024-03-25T19:43:19.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.877+0000 7f7d04b15640 10 monclient: init 2024-03-25T19:43:19.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:19.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:19.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 10 monclient(hunting): picked mon.noname-b con 0x7f7d000a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:19.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 10 monclient(hunting): picked mon.noname-c con 0x7f7d0013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:19.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 10 monclient(hunting): picked mon.noname-a con 0x7f7d001430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:19.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d00142520 con 0x7f7d001430c0 2024-03-25T19:43:19.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d000b28a0 con 0x7f7d0013f710 2024-03-25T19:43:19.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d000b2a30 con 0x7f7d000a98f0 2024-03-25T19:43:19.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:19.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.878+0000 7f7d04b15640 10 monclient(hunting): authenticate will time out at 5006.190261s 2024-03-25T19:43:19.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.882+0000 7f7cf7fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7d0013f710 0x7f7d001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:47526/0 (socket says 172.21.15.71:47526) 2024-03-25T19:43:19.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.882+0000 7f7cf7fff640 1 -- 172.21.15.71:0/2673147519 learned_addr learned my addr 172.21.15.71:0/2673147519 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:19.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.902+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498347710 0 0) 0x7f7d00142520 con 0x7f7d001430c0 2024-03-25T19:43:19.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.902+0000 7f7cfdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:19.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.902+0000 7f7cfdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:19.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.902+0000 7f7cfdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:19.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.902+0000 7f7cfdd9b640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:43:19.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.902+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cec003520 con 0x7f7d001430c0 2024-03-25T19:43:19.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2722031413 0 0) 0x7f7cec003520 con 0x7f7d001430c0 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 >> v1:172.21.15.71:6790/0 conn(0x7f7d000a98f0 legacy=0x7f7d000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 >> v1:172.21.15.193:6789/0 conn(0x7f7d0013f710 legacy=0x7f7d001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 1 monclient: found mon.noname-a 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d000b44c0 con 0x7f7d001430c0 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7d04b15640 5 monclient: authenticate success, global_id 4338 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7cf8001a10 con 0x7f7d001430c0 2024-03-25T19:43:19.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7d04b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7cfdd9b640 10 monclient: dump: 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:19.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.920+0000 7f7d04b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.980+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cf8001b70 con 0x7f7d001430c0 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.980+0000 7f7cfdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.980+0000 7f7cfdd9b640 1 -- 172.21.15.71:0/2673147519 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7cf8001fc0 con 0x7f7d001430c0 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.980+0000 7f7d04b15640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.980+0000 7f7d04b15640 4 set_mon_vals no callback set 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.980+0000 7f7cfdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.980+0000 7f7cfdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.980+0000 7f7cfdd9b640 10 monclient: dump: 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:19.980 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:19.981 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:19.981 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:19.981 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:19.981 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:19.981 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:19.981 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:19.981 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:19.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.981+0000 7f7d04b15640 10 monclient: shutdown 2024-03-25T19:43:19.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.981+0000 7f7d04b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.981+0000 7f7d04b15640 1 -- 172.21.15.71:0/2673147519 >> v1:172.21.15.71:6789/0 conn(0x7f7d001430c0 legacy=0x7f7d0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.981+0000 7f7d04b15640 1 -- 172.21.15.71:0/2673147519 shutdown_connections 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.981+0000 7f7d04b15640 1 -- 172.21.15.71:0/2673147519 >> 172.21.15.71:0/2673147519 conn(0x7f7d000b7ea0 msgr2=0x7f7d000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.981+0000 7f7d04b15640 1 -- 172.21.15.71:0/2673147519 shutdown_connections 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.981+0000 7f7d04b15640 1 -- 172.21.15.71:0/2673147519 wait complete. 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.982+0000 7f7d04b15640 10 monclient: build_initial_monmap 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.982+0000 7f7d04b15640 10 monclient: monmap: 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:19.983 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.982+0000 7f7d04b15640 1 librados: starting msgr at 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.982+0000 7f7d04b15640 1 librados: starting objecter 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.983+0000 7f7d04b15640 1 Processor -- start 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.983+0000 7f7d04b15640 1 -- start start 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.983+0000 7f7d04b15640 1 librados: setting wanted keys 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.983+0000 7f7d04b15640 1 librados: calling monclient init 2024-03-25T19:43:19.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.983+0000 7f7d04b15640 10 monclient: init 2024-03-25T19:43:19.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.983+0000 7f7d04b15640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:19.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.983+0000 7f7d04b15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:19.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.983+0000 7f7d04b15640 10 monclient(hunting): picked mon.noname-c con 0x7f7d000a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:19.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.984+0000 7f7d04b15640 10 monclient(hunting): picked mon.noname-a con 0x7f7d0013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:19.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.984+0000 7f7d04b15640 10 monclient(hunting): picked mon.noname-b con 0x7f7d001430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:19.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.984+0000 7f7d04b15640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d00139a10 con 0x7f7d0013f710 2024-03-25T19:43:19.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.984+0000 7f7d04b15640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d001d7880 con 0x7f7d001430c0 2024-03-25T19:43:19.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.984+0000 7f7d04b15640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d00137890 con 0x7f7d000a98f0 2024-03-25T19:43:19.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.984+0000 7f7d04b15640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:19.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.984+0000 7f7d04b15640 10 monclient(hunting): authenticate will time out at 5006.296060s 2024-03-25T19:43:19.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.991+0000 7f7cfed9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7d001430c0 0x7f7d001e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:47562/0 (socket says 172.21.15.71:47562) 2024-03-25T19:43:19.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.991+0000 7f7cfed9d640 1 -- 172.21.15.71:0/2270298594 learned_addr learned my addr 172.21.15.71:0/2270298594 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:19.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.995+0000 7f7cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f7d001430c0 2024-03-25T19:43:19.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:19.996+0000 7f7cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:20.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.002+0000 7f7cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f7d0013f710 2024-03-25T19:43:20.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.002+0000 7f7cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:20.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.005+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061949263 0 0) 0x7f7d001d7880 con 0x7f7d001430c0 2024-03-25T19:43:20.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.006+0000 7f7cf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:20.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.006+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:20.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.006+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:20.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.006+0000 7f7cf6ffd640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:43:20.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.006+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce0004eb0 con 0x7f7d001430c0 2024-03-25T19:43:20.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.012+0000 7f7cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f7d000a98f0 2024-03-25T19:43:20.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.012+0000 7f7cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33047500 0 0) 0x7f7ce0004eb0 con 0x7f7d001430c0 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d001d7880 con 0x7f7d001430c0 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7cf8001f40 con 0x7f7d001430c0 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:20.019 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759738689 0 0) 0x7f7d00137890 con 0x7f7d000a98f0 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 10 monclient(hunting): my global_id is 14348 2024-03-25T19:43:20.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.019+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce0004eb0 con 0x7f7d000a98f0 2024-03-25T19:43:20.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.020+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050347620 0 0) 0x7f7d00139a10 con 0x7f7d0013f710 2024-03-25T19:43:20.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.020+0000 7f7cf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:20.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.020+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:20.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.020+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:20.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.020+0000 7f7cf6ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:43:20.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.020+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d00137890 con 0x7f7d0013f710 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1168329093 0 0) 0x7f7d001d7880 con 0x7f7d001430c0 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 >> v1:172.21.15.71:6790/0 conn(0x7f7d000a98f0 legacy=0x7f7d001d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 >> v1:172.21.15.71:6789/0 conn(0x7f7d0013f710 legacy=0x7f7d00138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 1 monclient: found mon.b 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d001d84b0 con 0x7f7d001430c0 2024-03-25T19:43:20.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7cf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 10 monclient: _renew_subs 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d001d51f0 con 0x7f7d001430c0 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 10 monclient: _renew_subs 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d00138c60 con 0x7f7d001430c0 2024-03-25T19:43:20.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.027+0000 7f7d04b15640 1 librados: init done 2024-03-25T19:43:20.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.028+0000 7f7d04b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:20.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.028+0000 7f7d04b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:20.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.028+0000 7f7d04b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:20.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.028+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d000627b0 con 0x7f7d001430c0 2024-03-25T19:43:20.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.046+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cf8003c20 con 0x7f7d001430c0 2024-03-25T19:43:20.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.046+0000 7f7cf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:20.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.046+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7cf8004070 con 0x7f7d001430c0 2024-03-25T19:43:20.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.046+0000 7f7cf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:20.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.046+0000 7f7cf6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:20.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.046+0000 7f7cf6ffd640 10 monclient: dump: 2024-03-25T19:43:20.046 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:20.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.046+0000 7f7cfd59a640 4 set_mon_vals no callback set 2024-03-25T19:43:20.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.047+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f7cf8004290 con 0x7f7d001430c0 2024-03-25T19:43:20.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.047+0000 7f7cfe59c640 10 monclient: get_auth_request con 0x7f7ce006de80 auth_method 0 2024-03-25T19:43:20.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.047+0000 7f7cf6ffd640 10 client.4345.objecter ms_handle_connect 0x7f7ce006de80 2024-03-25T19:43:20.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.049+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f7cf8054c30 con 0x7f7d001430c0 2024-03-25T19:43:20.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.049+0000 7f7cf6ffd640 10 client.4345.objecter ms_dispatch 0x7f7d00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:20.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.049+0000 7f7cf6ffd640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:20.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.049+0000 7f7cf6ffd640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:20.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.049+0000 7f7cf6ffd640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-25T19:43:20.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.316+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cf8050740 con 0x7f7d001430c0 2024-03-25T19:43:20.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.316+0000 7f7cf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:20.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.316+0000 7f7cf6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:20.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.571+0000 7f7d04b15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:20.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.572+0000 7f7d04b15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:20.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.572+0000 7f7d04b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:20.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.572+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d0013df90 con 0x7f7d001430c0 2024-03-25T19:43:20.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.604+0000 7f7cf6ffd640 1 -- 172.21.15.71:0/2270298594 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f7cf8052670 con 0x7f7d001430c0 2024-03-25T19:43:20.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.604+0000 7f7cf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:20.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.604+0000 7f7cf6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:20.604 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:43:20.604 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:20.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 10 librados: watch_flush enter 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 10 librados: watch_flush exit 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 10 client.4345.objecter successfully canceled tick 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 >> v1:172.21.15.71:6800/3911735624 conn(0x7f7ce006de80 legacy=0x7f7ce0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 10 monclient: shutdown 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 >> v1:172.21.15.193:6789/0 conn(0x7f7d001430c0 legacy=0x7f7d001e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 shutdown_connections 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.606+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 >> 172.21.15.71:0/2270298594 conn(0x7f7d000b7ea0 msgr2=0x7f7d00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.607+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 shutdown_connections 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.607+0000 7f7d04b15640 1 -- 172.21.15.71:0/2270298594 wait complete. 2024-03-25T19:43:20.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:20.607+0000 7f7d04b15640 1 librados: shutdown 2024-03-25T19:43:20.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:27.425 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:27.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.543+0000 7fbd27a64640 10 monclient: get_monmap_and_config 2024-03-25T19:43:27.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.543+0000 7fbd27a64640 10 monclient: build_initial_monmap 2024-03-25T19:43:27.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.543+0000 7fbd27a64640 10 monclient: monmap: 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:43:27.543950+0000 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:43:27.543950+0000 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:43:27.543 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:43:27.544 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:27.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.544+0000 7fbd27a64640 1 Processor -- start 2024-03-25T19:43:27.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.544+0000 7fbd27a64640 1 -- start start 2024-03-25T19:43:27.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.544+0000 7fbd27a64640 10 monclient: init 2024-03-25T19:43:27.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.544+0000 7fbd27a64640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:27.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.544+0000 7fbd27a64640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:27.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.544+0000 7fbd27a64640 10 monclient(hunting): picked mon.noname-c con 0x7fbd200b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:27.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.545+0000 7fbd27a64640 10 monclient(hunting): picked mon.noname-a con 0x7fbd200b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:27.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.545+0000 7fbd27a64640 10 monclient(hunting): picked mon.noname-b con 0x7fbd2014be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:27.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.545+0000 7fbd27a64640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd200af940 con 0x7fbd200b1730 2024-03-25T19:43:27.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.545+0000 7fbd27a64640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd200b0a70 con 0x7fbd200b4d60 2024-03-25T19:43:27.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.545+0000 7fbd27a64640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd200b0d50 con 0x7fbd2014be90 2024-03-25T19:43:27.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.545+0000 7fbd27a64640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:27.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.545+0000 7fbd27a64640 10 monclient(hunting): authenticate will time out at 5013.856930s 2024-03-25T19:43:27.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.554+0000 7fbd24fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fbd200b1730 0x7fbd200b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:57924/0 (socket says 172.21.15.71:57924) 2024-03-25T19:43:27.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.554+0000 7fbd24fff640 1 -- 172.21.15.71:0/2702644598 learned_addr learned my addr 172.21.15.71:0/2702644598 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.563+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545916302 0 0) 0x7fbd200b0d50 con 0x7fbd2014be90 2024-03-25T19:43:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.563+0000 7fbd17fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.563+0000 7fbd17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.563+0000 7fbd17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.563+0000 7fbd17fff640 10 monclient(hunting): my global_id is 14351 2024-03-25T19:43:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.563+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd0c003520 con 0x7fbd2014be90 2024-03-25T19:43:27.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.573+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261878058 0 0) 0x7fbd200af940 con 0x7fbd200b1730 2024-03-25T19:43:27.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.573+0000 7fbd17fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.573+0000 7fbd17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.573+0000 7fbd17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.573+0000 7fbd17fff640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:43:27.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.573+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd200b0d50 con 0x7fbd200b1730 2024-03-25T19:43:27.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2277399075 0 0) 0x7fbd0c003520 con 0x7fbd2014be90 2024-03-25T19:43:27.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 >> v1:172.21.15.193:6789/0 conn(0x7fbd200b4d60 legacy=0x7fbd2013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 >> v1:172.21.15.71:6789/0 conn(0x7fbd200b1730 legacy=0x7fbd200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:27.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 1 monclient: found mon.noname-b 2024-03-25T19:43:27.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:27.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:43:27.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd200b1030 con 0x7fbd2014be90 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbd18001a10 con 0x7fbd2014be90 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd17fff640 10 monclient: dump: 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.583 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:27.584 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:27.584 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:27.584 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:27.584 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:27.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd27a64640 5 monclient: authenticate success, global_id 14351 2024-03-25T19:43:27.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.582+0000 7fbd27a64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:27.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.591+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd18001b70 con 0x7fbd2014be90 2024-03-25T19:43:27.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.591+0000 7fbd17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:27.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.591+0000 7fbd17fff640 1 -- 172.21.15.71:0/2702644598 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbd18001f80 con 0x7fbd2014be90 2024-03-25T19:43:27.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.591+0000 7fbd27a64640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.591+0000 7fbd27a64640 4 set_mon_vals no callback set 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.591+0000 7fbd17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.591+0000 7fbd17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.591+0000 7fbd17fff640 10 monclient: dump: 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:27.592 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:27.593 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:27.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.592+0000 7fbd27a64640 10 monclient: shutdown 2024-03-25T19:43:27.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.592+0000 7fbd27a64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:27.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.592+0000 7fbd27a64640 1 -- 172.21.15.71:0/2702644598 >> v1:172.21.15.71:6790/0 conn(0x7fbd2014be90 legacy=0x7fbd2014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.592+0000 7fbd27a64640 1 -- 172.21.15.71:0/2702644598 shutdown_connections 2024-03-25T19:43:27.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.592+0000 7fbd27a64640 1 -- 172.21.15.71:0/2702644598 >> 172.21.15.71:0/2702644598 conn(0x7fbd200617a0 msgr2=0x7fbd20061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:27.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.592+0000 7fbd27a64640 1 -- 172.21.15.71:0/2702644598 shutdown_connections 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.592+0000 7fbd27a64640 1 -- 172.21.15.71:0/2702644598 wait complete. 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.593+0000 7fbd27a64640 10 monclient: build_initial_monmap 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.593+0000 7fbd27a64640 10 monclient: monmap: 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:27.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.593+0000 7fbd27a64640 1 librados: starting msgr at 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.593+0000 7fbd27a64640 1 librados: starting objecter 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.593+0000 7fbd27a64640 1 Processor -- start 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 1 -- start start 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 1 librados: setting wanted keys 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 1 librados: calling monclient init 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 10 monclient: init 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 10 monclient(hunting): picked mon.noname-b con 0x7fbd200b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 10 monclient(hunting): picked mon.noname-c con 0x7fbd200b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 10 monclient(hunting): picked mon.noname-a con 0x7fbd2014be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.594+0000 7fbd27a64640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd20142060 con 0x7fbd2014be90 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.595+0000 7fbd27a64640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd201409a0 con 0x7fbd200b1730 2024-03-25T19:43:27.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.595+0000 7fbd27a64640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd20137cd0 con 0x7fbd200b4d60 2024-03-25T19:43:27.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.595+0000 7fbd27a64640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:27.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.595+0000 7fbd27a64640 10 monclient(hunting): authenticate will time out at 5013.906935s 2024-03-25T19:43:27.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.604+0000 7fbd24fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fbd200b4d60 0x7fbd201340d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:59516/0 (socket says 172.21.15.71:59516) 2024-03-25T19:43:27.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.604+0000 7fbd24fff640 1 -- 172.21.15.71:0/1580398267 learned_addr learned my addr 172.21.15.71:0/1580398267 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:27.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.605+0000 7fbd15ffb640 10 client.?.objecter ms_handle_connect 0x7fbd200b4d60 2024-03-25T19:43:27.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.605+0000 7fbd15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:27.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.615+0000 7fbd15ffb640 10 client.?.objecter ms_handle_connect 0x7fbd200b1730 2024-03-25T19:43:27.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.615+0000 7fbd15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:27.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.622+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978732747 0 0) 0x7fbd20137cd0 con 0x7fbd200b4d60 2024-03-25T19:43:27.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.622+0000 7fbd15ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.622+0000 7fbd15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.622+0000 7fbd15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.622+0000 7fbd15ffb640 10 monclient(hunting): my global_id is 14354 2024-03-25T19:43:27.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.622+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04004eb0 con 0x7fbd200b4d60 2024-03-25T19:43:27.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.625+0000 7fbd15ffb640 10 client.?.objecter ms_handle_connect 0x7fbd2014be90 2024-03-25T19:43:27.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.625+0000 7fbd15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:27.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.636+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009127416 0 0) 0x7fbd20142060 con 0x7fbd2014be90 2024-03-25T19:43:27.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.636+0000 7fbd15ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.636+0000 7fbd15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.636+0000 7fbd15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.636+0000 7fbd15ffb640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:43:27.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.636+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd20137cd0 con 0x7fbd2014be90 2024-03-25T19:43:27.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.638+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939460948 0 0) 0x7fbd201409a0 con 0x7fbd200b1730 2024-03-25T19:43:27.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.638+0000 7fbd15ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.638+0000 7fbd15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.638+0000 7fbd15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.638+0000 7fbd15ffb640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:43:27.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.638+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd20142060 con 0x7fbd200b1730 2024-03-25T19:43:27.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.653+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 788769166 0 0) 0x7fbd20142060 con 0x7fbd200b1730 2024-03-25T19:43:27.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.653+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd201409a0 con 0x7fbd200b1730 2024-03-25T19:43:27.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.653+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbd1c001870 con 0x7fbd200b1730 2024-03-25T19:43:27.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.653+0000 7fbd15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.653+0000 7fbd15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:27.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.653+0000 7fbd15ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:27.653 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:27.653 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:27.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.653+0000 7fbd15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:27.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.657+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 638774208 0 0) 0x7fbd20137cd0 con 0x7fbd2014be90 2024-03-25T19:43:27.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.657+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd20142060 con 0x7fbd2014be90 2024-03-25T19:43:27.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.657+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbd18001e80 con 0x7fbd2014be90 2024-03-25T19:43:27.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.657+0000 7fbd15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.657+0000 7fbd15ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:27.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.657+0000 7fbd15ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:27.657 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:27.658 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:27.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.664+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2193389235 0 0) 0x7fbd04004eb0 con 0x7fbd200b4d60 2024-03-25T19:43:27.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.664+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd20137cd0 con 0x7fbd200b4d60 2024-03-25T19:43:27.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.664+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbd100019b0 con 0x7fbd200b4d60 2024-03-25T19:43:27.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.664+0000 7fbd15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.664+0000 7fbd15ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.664+0000 7fbd15ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:27.665 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:27.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 864855817 0 0) 0x7fbd20142060 con 0x7fbd2014be90 2024-03-25T19:43:27.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 >> v1:172.21.15.71:6790/0 conn(0x7fbd200b4d60 legacy=0x7fbd201340d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 >> v1:172.21.15.193:6789/0 conn(0x7fbd200b1730 legacy=0x7fbd201375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:27.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 1 monclient: found mon.a 2024-03-25T19:43:27.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:27.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd20140e40 con 0x7fbd2014be90 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 10 monclient: _renew_subs 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd201347e0 con 0x7fbd2014be90 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 10 monclient: _renew_subs 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd20144e60 con 0x7fbd2014be90 2024-03-25T19:43:27.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.674+0000 7fbd27a64640 1 librados: init done 2024-03-25T19:43:27.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.675+0000 7fbd27a64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:27.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.675+0000 7fbd27a64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:27.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.675+0000 7fbd27a64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:27.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.675+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd20141050 con 0x7fbd2014be90 2024-03-25T19:43:27.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.968+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd180015a0 con 0x7fbd2014be90 2024-03-25T19:43:27.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.968+0000 7fbd15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:27.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.968+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fbd18003c20 con 0x7fbd2014be90 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.968+0000 7fbd15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.968+0000 7fbd15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.968+0000 7fbd15ffb640 10 monclient: dump: 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:27.969 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:27.970 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:27.970 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:27.970 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:27.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.968+0000 7fbd177fe640 4 set_mon_vals no callback set 2024-03-25T19:43:27.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.969+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fbd18019bf0 con 0x7fbd2014be90 2024-03-25T19:43:27.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.970+0000 7fbd25800640 10 monclient: get_auth_request con 0x7fbd0406e160 auth_method 0 2024-03-25T19:43:27.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.970+0000 7fbd15ffb640 10 client.4347.objecter ms_handle_connect 0x7fbd0406e160 2024-03-25T19:43:27.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.974+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fbd180537b0 con 0x7fbd2014be90 2024-03-25T19:43:27.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.974+0000 7fbd15ffb640 10 client.4347.objecter ms_dispatch 0x7fbd20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:27.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.974+0000 7fbd15ffb640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:27.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.974+0000 7fbd15ffb640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:27.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.975+0000 7fbd15ffb640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T19:43:27.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.975+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd1804f0f0 con 0x7fbd2014be90 2024-03-25T19:43:27.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.975+0000 7fbd15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:27.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:27.975+0000 7fbd15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:28.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.207+0000 7fbd27a64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:28.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.207+0000 7fbd27a64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:28.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.207+0000 7fbd27a64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:43:28.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.207+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd20141ea0 con 0x7fbd2014be90 2024-03-25T19:43:28.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.242+0000 7fbd15ffb640 1 -- 172.21.15.71:0/1580398267 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fbd180511f0 con 0x7fbd2014be90 2024-03-25T19:43:28.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.243+0000 7fbd15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:28.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.243+0000 7fbd15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:28.243 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:43:28.243 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:28.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 10 librados: watch_flush enter 2024-03-25T19:43:28.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 10 librados: watch_flush exit 2024-03-25T19:43:28.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T19:43:28.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 10 client.4347.objecter successfully canceled tick 2024-03-25T19:43:28.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 >> v1:172.21.15.71:6800/3911735624 conn(0x7fbd0406e160 legacy=0x7fbd04070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 10 monclient: shutdown 2024-03-25T19:43:28.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:28.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 >> v1:172.21.15.71:6789/0 conn(0x7fbd2014be90 legacy=0x7fbd201e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 shutdown_connections 2024-03-25T19:43:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 >> 172.21.15.71:0/1580398267 conn(0x7fbd200617a0 msgr2=0x7fbd200b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 shutdown_connections 2024-03-25T19:43:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 1 -- 172.21.15.71:0/1580398267 wait complete. 2024-03-25T19:43:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:28.245+0000 7fbd27a64640 1 librados: shutdown 2024-03-25T19:43:28.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:35.065 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.178+0000 7fa33e0a8640 10 monclient: get_monmap_and_config 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.178+0000 7fa33e0a8640 10 monclient: build_initial_monmap 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.178+0000 7fa33e0a8640 10 monclient: monmap: 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:43:35.179360+0000 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:43:35.179360+0000 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:43:35.178 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:35.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.179+0000 7fa33e0a8640 1 Processor -- start 2024-03-25T19:43:35.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.179+0000 7fa33e0a8640 1 -- start start 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.179+0000 7fa33e0a8640 10 monclient: init 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa3380b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa3380b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa33814be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380af940 con 0x7fa33814be90 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380b0a70 con 0x7fa3380b1730 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380b0d50 con 0x7fa3380b4d60 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:35.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.180+0000 7fa33e0a8640 10 monclient(hunting): authenticate will time out at 5021.491980s 2024-03-25T19:43:35.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.195+0000 7fa336ffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa3380b1730 0x7fa3380b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:40402/0 (socket says 172.21.15.71:40402) 2024-03-25T19:43:35.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.195+0000 7fa336ffd640 1 -- 172.21.15.71:0/2073495290 learned_addr learned my addr 172.21.15.71:0/2073495290 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:35.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.207+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163813468 0 0) 0x7fa3380b0a70 con 0x7fa3380b1730 2024-03-25T19:43:35.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.207+0000 7fa3367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.207+0000 7fa3367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.207+0000 7fa3367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.207+0000 7fa3367fc640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:43:35.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.207+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa320003520 con 0x7fa3380b1730 2024-03-25T19:43:35.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.208+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579005388 0 0) 0x7fa3380af940 con 0x7fa33814be90 2024-03-25T19:43:35.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.208+0000 7fa3367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.208+0000 7fa3367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.208+0000 7fa3367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.208+0000 7fa3367fc640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:43:35.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.208+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3380b0a70 con 0x7fa33814be90 2024-03-25T19:43:35.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.216+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251048808 0 0) 0x7fa3380b0d50 con 0x7fa3380b4d60 2024-03-25T19:43:35.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.216+0000 7fa3367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.216+0000 7fa3367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.216+0000 7fa3367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.216+0000 7fa3367fc640 10 monclient(hunting): my global_id is 14357 2024-03-25T19:43:35.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.216+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3380af940 con 0x7fa3380b4d60 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 754282359 0 0) 0x7fa3380b0a70 con 0x7fa33814be90 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 >> v1:172.21.15.71:6790/0 conn(0x7fa3380b4d60 legacy=0x7fa33813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 >> v1:172.21.15.193:6789/0 conn(0x7fa3380b1730 legacy=0x7fa3380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 1 monclient: found mon.noname-a 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3380b1030 con 0x7fa33814be90 2024-03-25T19:43:35.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa3300019f0 con 0x7fa33814be90 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa33e0a8640 5 monclient: authenticate success, global_id 4350 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa33e0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa3367fc640 10 monclient: dump: 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:35.221 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:35.222 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:35.222 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:35.222 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:35.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.220+0000 7fa33e0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.244+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa330001b50 con 0x7fa33814be90 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.244+0000 7fa3367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.244+0000 7fa3367fc640 1 -- 172.21.15.71:0/2073495290 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa330001f60 con 0x7fa33814be90 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.244+0000 7fa3367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.244+0000 7fa3367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.244+0000 7fa3367fc640 10 monclient: dump: 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:35.244 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:35.245 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:35.245 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:35.245 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:35.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.244+0000 7fa33e0a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:35.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.244+0000 7fa33e0a8640 4 set_mon_vals no callback set 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.245+0000 7fa33e0a8640 10 monclient: shutdown 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.245+0000 7fa33e0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.245+0000 7fa33e0a8640 1 -- 172.21.15.71:0/2073495290 >> v1:172.21.15.71:6789/0 conn(0x7fa33814be90 legacy=0x7fa33814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.245+0000 7fa33e0a8640 1 -- 172.21.15.71:0/2073495290 shutdown_connections 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.245+0000 7fa33e0a8640 1 -- 172.21.15.71:0/2073495290 >> 172.21.15.71:0/2073495290 conn(0x7fa3380617a0 msgr2=0x7fa338061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.245+0000 7fa33e0a8640 1 -- 172.21.15.71:0/2073495290 shutdown_connections 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.245+0000 7fa33e0a8640 1 -- 172.21.15.71:0/2073495290 wait complete. 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.246+0000 7fa33e0a8640 10 monclient: build_initial_monmap 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.246+0000 7fa33e0a8640 10 monclient: monmap: 2024-03-25T19:43:35.246 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:43:35.247 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.247+0000 7fa33e0a8640 1 librados: starting msgr at 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.247+0000 7fa33e0a8640 1 librados: starting objecter 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.247+0000 7fa33e0a8640 1 Processor -- start 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.247+0000 7fa33e0a8640 1 -- start start 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.247+0000 7fa33e0a8640 1 librados: setting wanted keys 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.247+0000 7fa33e0a8640 1 librados: calling monclient init 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.247+0000 7fa33e0a8640 10 monclient: init 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa3380b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa3380b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:35.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa33814be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:35.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33813d0f0 con 0x7fa3380b4d60 2024-03-25T19:43:35.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa338141730 con 0x7fa3380b1730 2024-03-25T19:43:35.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3381431a0 con 0x7fa33814be90 2024-03-25T19:43:35.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:35.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.248+0000 7fa33e0a8640 10 monclient(hunting): authenticate will time out at 5021.560114s 2024-03-25T19:43:35.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.253+0000 7fa337fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa33814be90 0x7fa3381e2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35926/0 (socket says 172.21.15.71:35926) 2024-03-25T19:43:35.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.253+0000 7fa337fff640 1 -- 172.21.15.71:0/3014031734 learned_addr learned my addr 172.21.15.71:0/3014031734 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:35.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.254+0000 7fa31ffff640 10 client.?.objecter ms_handle_connect 0x7fa33814be90 2024-03-25T19:43:35.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.254+0000 7fa31ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:35.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.256+0000 7fa31ffff640 10 client.?.objecter ms_handle_connect 0x7fa3380b1730 2024-03-25T19:43:35.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.256+0000 7fa31ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:35.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.264+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177003026 0 0) 0x7fa338141730 con 0x7fa3380b1730 2024-03-25T19:43:35.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.264+0000 7fa31ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.264+0000 7fa31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.264+0000 7fa31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.264+0000 7fa31ffff640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:43:35.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.264+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324004eb0 con 0x7fa3380b1730 2024-03-25T19:43:35.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.265+0000 7fa31ffff640 10 client.?.objecter ms_handle_connect 0x7fa3380b4d60 2024-03-25T19:43:35.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.265+0000 7fa31ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:35.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.271+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921423810 0 0) 0x7fa3381431a0 con 0x7fa33814be90 2024-03-25T19:43:35.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.271+0000 7fa31ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.271+0000 7fa31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.271+0000 7fa31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.271+0000 7fa31ffff640 10 monclient(hunting): my global_id is 14360 2024-03-25T19:43:35.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.271+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa338141730 con 0x7fa33814be90 2024-03-25T19:43:35.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.274+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205459699 0 0) 0x7fa33813d0f0 con 0x7fa3380b4d60 2024-03-25T19:43:35.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.274+0000 7fa31ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.274+0000 7fa31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.274+0000 7fa31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.274+0000 7fa31ffff640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:43:35.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.274+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3381431a0 con 0x7fa3380b4d60 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.282+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1018209888 0 0) 0x7fa324004eb0 con 0x7fa3380b1730 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.282+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa33813d0f0 con 0x7fa3380b1730 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.282+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa3280018f0 con 0x7fa3380b1730 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.282+0000 7fa31ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.282+0000 7fa31ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.282+0000 7fa31ffff640 10 monclient(hunting): dump: 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:35.282 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:35.283 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:35.283 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:35.283 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:35.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.282+0000 7fa31ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4193005816 0 0) 0x7fa33813d0f0 con 0x7fa3380b1730 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 >> v1:172.21.15.71:6790/0 conn(0x7fa33814be90 legacy=0x7fa3381e2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 >> v1:172.21.15.71:6789/0 conn(0x7fa3380b4d60 legacy=0x7fa3381d7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 1 monclient: found mon.b 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa338143330 con 0x7fa3380b1730 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 10 monclient: _finish_auth 0 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa31ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:35.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 5 monclient: authenticate success, global_id 4357 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 10 monclient: _renew_subs 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa338140380 con 0x7fa3380b1730 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 10 monclient: _renew_subs 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3381d8280 con 0x7fa3380b1730 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.293+0000 7fa33e0a8640 1 librados: init done 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.294+0000 7fa33e0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:35.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.294+0000 7fa33e0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:35.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.294+0000 7fa33e0a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:35.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.294+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3381d8580 con 0x7fa3380b1730 2024-03-25T19:43:35.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.584+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa328003610 con 0x7fa3380b1730 2024-03-25T19:43:35.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.584+0000 7fa31ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:35.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.584+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa3280039a0 con 0x7fa3380b1730 2024-03-25T19:43:35.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.584+0000 7fa31ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.584+0000 7fa31ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.584+0000 7fa31ffff640 10 monclient: dump: 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.584+0000 7fa335ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:35.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.585+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fa328003bc0 con 0x7fa3380b1730 2024-03-25T19:43:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.586+0000 7fa336ffd640 10 monclient: get_auth_request con 0x7fa32406ddf0 auth_method 0 2024-03-25T19:43:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.586+0000 7fa31ffff640 10 client.4357.objecter ms_handle_connect 0x7fa32406ddf0 2024-03-25T19:43:35.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.590+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fa3280535e0 con 0x7fa3380b1730 2024-03-25T19:43:35.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.590+0000 7fa31ffff640 10 client.4357.objecter ms_dispatch 0x7fa338000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:35.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.590+0000 7fa31ffff640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:35.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.590+0000 7fa31ffff640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:35.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.590+0000 7fa31ffff640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-25T19:43:35.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.590+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa32804f0f0 con 0x7fa3380b1730 2024-03-25T19:43:35.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.590+0000 7fa31ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:35.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.590+0000 7fa31ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:35.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.842+0000 7fa33e0a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:35.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.842+0000 7fa33e0a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:35.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.842+0000 7fa33e0a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:35.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.842+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3380627b0 con 0x7fa3380b1730 2024-03-25T19:43:35.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.972+0000 7fa31ffff640 1 -- 172.21.15.71:0/3014031734 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fa328051020 con 0x7fa3380b1730 2024-03-25T19:43:35.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.972+0000 7fa31ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:35.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.972+0000 7fa31ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:35.972 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:43:35.972 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 10 librados: watch_flush enter 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 10 librados: watch_flush exit 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 10 client.4357.objecter successfully canceled tick 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 >> v1:172.21.15.71:6800/3911735624 conn(0x7fa32406ddf0 legacy=0x7fa3240702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 10 monclient: shutdown 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 >> v1:172.21.15.193:6789/0 conn(0x7fa3380b1730 legacy=0x7fa338141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 shutdown_connections 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 >> 172.21.15.71:0/3014031734 conn(0x7fa3380617a0 msgr2=0x7fa33813c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:35.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 shutdown_connections 2024-03-25T19:43:35.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 1 -- 172.21.15.71:0/3014031734 wait complete. 2024-03-25T19:43:35.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:35.975+0000 7fa33e0a8640 1 librados: shutdown 2024-03-25T19:43:35.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:36.679 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:19.435 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.686 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:19.443 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.728 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:19.484 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.736 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:19.456 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.744 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:19.463 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.748 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:19.505 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.761 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:19.517 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.776 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:19.496 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.786 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:19.543 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:42.792 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:42.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.910+0000 7ff5826a1640 10 monclient: get_monmap_and_config 2024-03-25T19:43:42.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.910+0000 7ff5826a1640 10 monclient: build_initial_monmap 2024-03-25T19:43:42.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.910+0000 7ff5826a1640 10 monclient: monmap: 2024-03-25T19:43:42.910 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:43:42.911722+0000 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:43:42.911722+0000 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:43:42.911 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:42.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.911+0000 7ff5826a1640 1 Processor -- start 2024-03-25T19:43:42.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.911+0000 7ff5826a1640 1 -- start start 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.911+0000 7ff5826a1640 10 monclient: init 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 10 monclient(hunting): picked mon.noname-a con 0x7ff57c0a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 10 monclient(hunting): picked mon.noname-b con 0x7ff57c13f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 10 monclient(hunting): picked mon.noname-c con 0x7ff57c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c142520 con 0x7ff57c0a98f0 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c0b28a0 con 0x7ff57c1430c0 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c0b2a30 con 0x7ff57c13f710 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:42.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.912+0000 7ff5826a1640 10 monclient(hunting): authenticate will time out at 5029.224326s 2024-03-25T19:43:42.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.929+0000 7ff580c3e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff57c1430c0 0x7ff57c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:57752/0 (socket says 172.21.15.71:57752) 2024-03-25T19:43:42.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.929+0000 7ff580c3e640 1 -- 172.21.15.71:0/4024857678 learned_addr learned my addr 172.21.15.71:0/4024857678 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.946+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336508489 0 0) 0x7ff57c0b28a0 con 0x7ff57c1430c0 2024-03-25T19:43:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.946+0000 7ff57affd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.946+0000 7ff57affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.946+0000 7ff57affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.946+0000 7ff57affd640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:43:42.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.946+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff564003520 con 0x7ff57c1430c0 2024-03-25T19:43:42.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.952+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957133132 0 0) 0x7ff57c142520 con 0x7ff57c0a98f0 2024-03-25T19:43:42.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.952+0000 7ff57affd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:42.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.952+0000 7ff57affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:42.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.952+0000 7ff57affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:42.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.952+0000 7ff57affd640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:43:42.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.952+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57c0b28a0 con 0x7ff57c0a98f0 2024-03-25T19:43:42.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.954+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689491227 0 0) 0x7ff57c0b2a30 con 0x7ff57c13f710 2024-03-25T19:43:42.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.954+0000 7ff57affd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:42.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.954+0000 7ff57affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:42.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.954+0000 7ff57affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:42.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.954+0000 7ff57affd640 10 monclient(hunting): my global_id is 14363 2024-03-25T19:43:42.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.954+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57c142520 con 0x7ff57c13f710 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 674179118 0 0) 0x7ff564003520 con 0x7ff57c1430c0 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 >> v1:172.21.15.71:6790/0 conn(0x7ff57c13f710 legacy=0x7ff57c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 >> v1:172.21.15.71:6789/0 conn(0x7ff57c0a98f0 legacy=0x7ff57c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 1 monclient: found mon.noname-c 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c0b44c0 con 0x7ff57c1430c0 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 10 monclient: _finish_auth 0 2024-03-25T19:43:42.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff574001a10 con 0x7ff57c1430c0 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff5826a1640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff5826a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff57affd640 10 monclient: dump: 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:42.972 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:42.973 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:42.973 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:42.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.971+0000 7ff5826a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:42.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff574001b70 con 0x7ff57c1430c0 2024-03-25T19:43:42.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff57affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:42.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff57affd640 1 -- 172.21.15.71:0/4024857678 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff574001f80 con 0x7ff57c1430c0 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff57affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff57affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff57affd640 10 monclient: dump: 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:42.991 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:42.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff5826a1640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:42.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff5826a1640 4 set_mon_vals no callback set 2024-03-25T19:43:42.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff5826a1640 10 monclient: shutdown 2024-03-25T19:43:42.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff5826a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:42.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.991+0000 7ff5826a1640 1 -- 172.21.15.71:0/4024857678 >> v1:172.21.15.193:6789/0 conn(0x7ff57c1430c0 legacy=0x7ff57c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:42.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.992+0000 7ff5826a1640 1 -- 172.21.15.71:0/4024857678 shutdown_connections 2024-03-25T19:43:42.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.992+0000 7ff5826a1640 1 -- 172.21.15.71:0/4024857678 >> 172.21.15.71:0/4024857678 conn(0x7ff57c0b7ea0 msgr2=0x7ff57c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:42.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.992+0000 7ff5826a1640 1 -- 172.21.15.71:0/4024857678 shutdown_connections 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.992+0000 7ff5826a1640 1 -- 172.21.15.71:0/4024857678 wait complete. 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.992+0000 7ff5826a1640 10 monclient: build_initial_monmap 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.992+0000 7ff5826a1640 10 monclient: monmap: 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:43:42.995 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 1 librados: starting msgr at 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 1 librados: starting objecter 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 1 Processor -- start 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 1 -- start start 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 1 librados: setting wanted keys 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 1 librados: calling monclient init 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 10 monclient: init 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.993+0000 7ff5826a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.994+0000 7ff5826a1640 10 monclient(hunting): picked mon.noname-c con 0x7ff57c0a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.994+0000 7ff5826a1640 10 monclient(hunting): picked mon.noname-a con 0x7ff57c13f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.994+0000 7ff5826a1640 10 monclient(hunting): picked mon.noname-b con 0x7ff57c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:42.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.994+0000 7ff5826a1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c1369d0 con 0x7ff57c13f710 2024-03-25T19:43:42.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.994+0000 7ff5826a1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c133a60 con 0x7ff57c1430c0 2024-03-25T19:43:42.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.994+0000 7ff5826a1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c0ab9e0 con 0x7ff57c0a98f0 2024-03-25T19:43:42.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.994+0000 7ff5826a1640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:42.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:42.994+0000 7ff5826a1640 10 monclient(hunting): authenticate will time out at 5029.305985s 2024-03-25T19:43:43.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.002+0000 7ff57b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff57c13f710 0x7ff57c1362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58646/0 (socket says 172.21.15.71:58646) 2024-03-25T19:43:43.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.002+0000 7ff57b7fe640 1 -- 172.21.15.71:0/3236428448 learned_addr learned my addr 172.21.15.71:0/3236428448 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:43.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.005+0000 7ff578ff9640 10 client.?.objecter ms_handle_connect 0x7ff57c13f710 2024-03-25T19:43:43.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.005+0000 7ff578ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:43.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.018+0000 7ff578ff9640 10 client.?.objecter ms_handle_connect 0x7ff57c1430c0 2024-03-25T19:43:43.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.018+0000 7ff578ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:43.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.022+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704862087 0 0) 0x7ff57c1369d0 con 0x7ff57c13f710 2024-03-25T19:43:43.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.022+0000 7ff578ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:43.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.022+0000 7ff578ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:43.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.022+0000 7ff578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:43.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.022+0000 7ff578ff9640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:43:43.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.022+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff568004eb0 con 0x7ff57c13f710 2024-03-25T19:43:43.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.024+0000 7ff578ff9640 10 client.?.objecter ms_handle_connect 0x7ff57c0a98f0 2024-03-25T19:43:43.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.024+0000 7ff578ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:43.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.032+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145363138 0 0) 0x7ff57c0ab9e0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.032+0000 7ff578ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:43.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.032+0000 7ff578ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:43.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.032+0000 7ff578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:43.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.032+0000 7ff578ff9640 10 monclient(hunting): my global_id is 14366 2024-03-25T19:43:43.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.032+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57c1369d0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.043+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 188617919 0 0) 0x7ff568004eb0 con 0x7ff57c13f710 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.043+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff57c0ab9e0 con 0x7ff57c13f710 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.043+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff570001a10 con 0x7ff57c13f710 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.043+0000 7ff578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.043+0000 7ff578ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.043+0000 7ff578ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:43.043 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:43.044 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:43.044 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:43.044 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:43.044 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:43.044 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:43.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.043+0000 7ff578ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:43.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.047+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070045230 0 0) 0x7ff57c1369d0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.047+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff568004eb0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.047+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff56c001970 con 0x7ff57c0a98f0 2024-03-25T19:43:43.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.047+0000 7ff578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:43.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.047+0000 7ff578ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:43.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.047+0000 7ff578ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.048+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017095377 0 0) 0x7ff57c133a60 con 0x7ff57c1430c0 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.048+0000 7ff578ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.048+0000 7ff578ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.048+0000 7ff578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.048+0000 7ff578ff9640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.048+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57c1369d0 con 0x7ff57c1430c0 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.057+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1907214842 0 0) 0x7ff568004eb0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.057+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 >> v1:172.21.15.193:6789/0 conn(0x7ff57c1430c0 legacy=0x7ff57c133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 >> v1:172.21.15.71:6789/0 conn(0x7ff57c13f710 legacy=0x7ff57c1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 1 monclient: found mon.c 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c0abcc0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff578ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 5 monclient: authenticate success, global_id 14366 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 10 monclient: _renew_subs 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c0abfa0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 10 client.14366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 10 monclient: _renew_subs 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c137e40 con 0x7ff57c0a98f0 2024-03-25T19:43:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.058+0000 7ff5826a1640 1 librados: init done 2024-03-25T19:43:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.059+0000 7ff5826a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.059+0000 7ff5826a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.059+0000 7ff5826a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.059+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c0627b0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.351+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff56c003610 con 0x7ff57c0a98f0 2024-03-25T19:43:43.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.351+0000 7ff578ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.351+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff56c003920 con 0x7ff57c0a98f0 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.351+0000 7ff578ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.351+0000 7ff578ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.351+0000 7ff578ff9640 10 monclient: dump: 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:43.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.351+0000 7ff57a7fc640 4 set_mon_vals no callback set 2024-03-25T19:43:43.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.352+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7ff56c0198f0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.353+0000 7ff57b7fe640 10 monclient: get_auth_request con 0x7ff56806df50 auth_method 0 2024-03-25T19:43:43.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.353+0000 7ff578ff9640 10 client.14366.objecter ms_handle_connect 0x7ff56806df50 2024-03-25T19:43:43.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.358+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7ff56c053570 con 0x7ff57c0a98f0 2024-03-25T19:43:43.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.358+0000 7ff578ff9640 10 client.14366.objecter ms_dispatch 0x7ff57c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:43.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.358+0000 7ff578ff9640 3 client.14366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:43.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.358+0000 7ff578ff9640 3 client.14366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:43.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.358+0000 7ff578ff9640 20 client.14366.objecter dump_active .. 0 homeless 2024-03-25T19:43:43.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.358+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff56c04f080 con 0x7ff57c0a98f0 2024-03-25T19:43:43.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.358+0000 7ff578ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:43.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.358+0000 7ff578ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:43.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.607+0000 7ff5826a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:43.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.607+0000 7ff5826a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:43.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.607+0000 7ff5826a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:43.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.607+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff57c13dfd0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.632+0000 7ff578ff9640 1 -- 172.21.15.71:0/3236428448 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7ff56c050fb0 con 0x7ff57c0a98f0 2024-03-25T19:43:43.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.632+0000 7ff578ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:43.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.632+0000 7ff578ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:43.632 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:43:43.632 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:43.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.634+0000 7ff5826a1640 10 librados: watch_flush enter 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.634+0000 7ff5826a1640 10 librados: watch_flush exit 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.634+0000 7ff5826a1640 20 client.14366.objecter shutdown clearing up homeless session... 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.634+0000 7ff5826a1640 10 client.14366.objecter successfully canceled tick 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.634+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 >> v1:172.21.15.71:6800/3911735624 conn(0x7ff56806df50 legacy=0x7ff568070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.634+0000 7ff5826a1640 10 monclient: shutdown 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.634+0000 7ff5826a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.634+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 >> v1:172.21.15.71:6790/0 conn(0x7ff57c0a98f0 legacy=0x7ff57c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.635+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 shutdown_connections 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.635+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 >> 172.21.15.71:0/3236428448 conn(0x7ff57c0b7ea0 msgr2=0x7ff57c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.635+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 shutdown_connections 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.635+0000 7ff5826a1640 1 -- 172.21.15.71:0/3236428448 wait complete. 2024-03-25T19:43:43.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:43.635+0000 7ff5826a1640 1 librados: shutdown 2024-03-25T19:43:43.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:50.452 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:50.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.567+0000 7f6a5fdc6640 10 monclient: get_monmap_and_config 2024-03-25T19:43:50.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.567+0000 7f6a5fdc6640 10 monclient: build_initial_monmap 2024-03-25T19:43:50.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.567+0000 7f6a5fdc6640 10 monclient: monmap: 2024-03-25T19:43:50.568 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:50.568 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:50.569 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:43:50.569769+0000 2024-03-25T19:43:50.569 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:43:50.569769+0000 2024-03-25T19:43:50.569 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:50.569 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:50.569 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:50.569 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:43:50.569 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:43:50.569 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:50.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.568+0000 7f6a5fdc6640 1 Processor -- start 2024-03-25T19:43:50.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.568+0000 7f6a5fdc6640 1 -- start start 2024-03-25T19:43:50.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.568+0000 7f6a5fdc6640 10 monclient: init 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f6a580b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f6a580b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f6a5814be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a580af940 con 0x7f6a5814be90 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a580b0a70 con 0x7f6a580b1730 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a580b0d50 con 0x7f6a580b4d60 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:50.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.569+0000 7f6a5fdc6640 10 monclient(hunting): authenticate will time out at 5036.882333s 2024-03-25T19:43:50.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.580+0000 7f6a5d361640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f6a580b1730 0x7f6a580b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:45238/0 (socket says 172.21.15.71:45238) 2024-03-25T19:43:50.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.580+0000 7f6a5d361640 1 -- 172.21.15.71:0/244603330 learned_addr learned my addr 172.21.15.71:0/244603330 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:50.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.596+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369182857 0 0) 0x7f6a580b0a70 con 0x7f6a580b1730 2024-03-25T19:43:50.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.597+0000 7f6a5cb60640 10 monclient(hunting): _negotiate 2024-03-25T19:43:50.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.597+0000 7f6a5cb60640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:50.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.597+0000 7f6a5cb60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:50.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.597+0000 7f6a5cb60640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:43:50.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.597+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a44003520 con 0x7f6a580b1730 2024-03-25T19:43:50.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.601+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977226099 0 0) 0x7f6a580b0d50 con 0x7f6a580b4d60 2024-03-25T19:43:50.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.601+0000 7f6a5cb60640 10 monclient(hunting): _negotiate 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.601+0000 7f6a5cb60640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.601+0000 7f6a5cb60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.601+0000 7f6a5cb60640 10 monclient(hunting): my global_id is 14369 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.601+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a580b0a70 con 0x7f6a580b4d60 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.602+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705977515 0 0) 0x7f6a580af940 con 0x7f6a5814be90 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.602+0000 7f6a5cb60640 10 monclient(hunting): _negotiate 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.602+0000 7f6a5cb60640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.602+0000 7f6a5cb60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.602+0000 7f6a5cb60640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:43:50.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.602+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a580b0d50 con 0x7f6a5814be90 2024-03-25T19:43:50.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 45364741 0 0) 0x7f6a580b0a70 con 0x7f6a580b4d60 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 >> v1:172.21.15.193:6789/0 conn(0x7f6a580b1730 legacy=0x7f6a580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 >> v1:172.21.15.71:6789/0 conn(0x7f6a5814be90 legacy=0x7f6a5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 10 monclient: _finish_hunting 0 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 1 monclient: found mon.noname-b 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a580b1030 con 0x7f6a580b4d60 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 10 monclient: _finish_auth 0 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 10 monclient: _check_auth_tickets 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5cb60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5fdc6640 5 monclient: authenticate success, global_id 14369 2024-03-25T19:43:50.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.609+0000 7f6a5fdc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:50.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.612+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a54001630 con 0x7f6a580b4d60 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.612+0000 7f6a5cb60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.612+0000 7f6a5cb60640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.612+0000 7f6a5cb60640 10 monclient: dump: 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:50.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.612+0000 7f6a5fdc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:50.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.630+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a54001880 con 0x7f6a580b4d60 2024-03-25T19:43:50.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.630+0000 7f6a5cb60640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:50.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.630+0000 7f6a5cb60640 1 -- 172.21.15.71:0/244603330 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a54001c10 con 0x7f6a580b4d60 2024-03-25T19:43:50.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.630+0000 7f6a5cb60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.630+0000 7f6a5cb60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:50.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.630+0000 7f6a5cb60640 10 monclient: dump: 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.630+0000 7f6a5fdc6640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:50.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.630+0000 7f6a5fdc6640 4 set_mon_vals no callback set 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.631+0000 7f6a5fdc6640 10 monclient: shutdown 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.631+0000 7f6a5fdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.631+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/244603330 >> v1:172.21.15.71:6790/0 conn(0x7f6a580b4d60 legacy=0x7f6a5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.631+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/244603330 shutdown_connections 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.631+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/244603330 >> 172.21.15.71:0/244603330 conn(0x7f6a580617a0 msgr2=0x7f6a58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.631+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/244603330 shutdown_connections 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.631+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/244603330 wait complete. 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 10 monclient: build_initial_monmap 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 10 monclient: monmap: 2024-03-25T19:43:50.635 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 1 librados: starting msgr at 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 1 librados: starting objecter 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 1 Processor -- start 2024-03-25T19:43:50.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 1 -- start start 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 1 librados: setting wanted keys 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 1 librados: calling monclient init 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.632+0000 7f6a5fdc6640 10 monclient: init 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f6a580b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f6a580b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f6a5814be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a58135b10 con 0x7f6a580b4d60 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a58134b70 con 0x7f6a5814be90 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5813bb10 con 0x7f6a580b1730 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:50.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.633+0000 7f6a5fdc6640 10 monclient(hunting): authenticate will time out at 5036.946405s 2024-03-25T19:43:50.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.646+0000 7f6a5db62640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f6a580b1730 0x7f6a58135400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51628/0 (socket says 172.21.15.71:51628) 2024-03-25T19:43:50.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.646+0000 7f6a5db62640 1 -- 172.21.15.71:0/3571199798 learned_addr learned my addr 172.21.15.71:0/3571199798 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:50.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.647+0000 7f6a4e7fc640 10 client.?.objecter ms_handle_connect 0x7f6a580b1730 2024-03-25T19:43:50.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.647+0000 7f6a4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:50.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.660+0000 7f6a4e7fc640 10 client.?.objecter ms_handle_connect 0x7f6a5814be90 2024-03-25T19:43:50.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.660+0000 7f6a4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:50.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.662+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675407025 0 0) 0x7f6a5813bb10 con 0x7f6a580b1730 2024-03-25T19:43:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.662+0000 7f6a4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.662+0000 7f6a4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.662+0000 7f6a4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.662+0000 7f6a4e7fc640 10 monclient(hunting): my global_id is 14372 2024-03-25T19:43:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.662+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a3c004eb0 con 0x7f6a580b1730 2024-03-25T19:43:50.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.665+0000 7f6a4e7fc640 10 client.?.objecter ms_handle_connect 0x7f6a580b4d60 2024-03-25T19:43:50.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.665+0000 7f6a4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:50.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.670+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3115236525 0 0) 0x7f6a3c004eb0 con 0x7f6a580b1730 2024-03-25T19:43:50.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.670+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a5813bb10 con 0x7f6a580b1730 2024-03-25T19:43:50.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.677+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219229279 0 0) 0x7f6a58135b10 con 0x7f6a580b4d60 2024-03-25T19:43:50.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.677+0000 7f6a4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:50.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.677+0000 7f6a4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:50.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.677+0000 7f6a4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:50.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.677+0000 7f6a4e7fc640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:43:50.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.677+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a3c004eb0 con 0x7f6a580b4d60 2024-03-25T19:43:50.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a540035f0 con 0x7f6a580b1730 2024-03-25T19:43:50.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:50.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 10 monclient(hunting): dump: 2024-03-25T19:43:50.679 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2359005213 0 0) 0x7f6a5813bb10 con 0x7f6a580b1730 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 >> v1:172.21.15.193:6789/0 conn(0x7f6a5814be90 legacy=0x7f6a581db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 >> v1:172.21.15.71:6789/0 conn(0x7f6a580b4d60 legacy=0x7f6a581dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 1 monclient: found mon.c 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a581dbb50 con 0x7f6a580b1730 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a5fdc6640 5 monclient: authenticate success, global_id 14372 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a5fdc6640 10 monclient: _renew_subs 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a5fdc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5813b540 con 0x7f6a580b1730 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a5fdc6640 10 client.14372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a5fdc6640 10 monclient: _renew_subs 2024-03-25T19:43:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a5fdc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:50.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.678+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a581330c0 con 0x7f6a580b1730 2024-03-25T19:43:50.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.679+0000 7f6a5fdc6640 1 librados: init done 2024-03-25T19:43:50.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.680+0000 7f6a5fdc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:50.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.680+0000 7f6a5fdc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:50.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.680+0000 7f6a5fdc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:50.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.680+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a58135070 con 0x7f6a580b1730 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.979+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a54003ac0 con 0x7f6a580b1730 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.979+0000 7f6a4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.979+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a54003f10 con 0x7f6a580b1730 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.979+0000 7f6a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.979+0000 7f6a4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.979+0000 7f6a4e7fc640 10 monclient: dump: 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.980 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:50.981 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:50.981 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:50.981 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:50.981 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:50.981 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.979+0000 7f6a4ffff640 4 set_mon_vals no callback set 2024-03-25T19:43:50.981 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.980+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f6a54004130 con 0x7f6a580b1730 2024-03-25T19:43:50.981 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.980+0000 7f6a5d361640 10 monclient: get_auth_request con 0x7f6a3c06dd00 auth_method 0 2024-03-25T19:43:50.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.980+0000 7f6a4e7fc640 10 client.14372.objecter ms_handle_connect 0x7f6a3c06dd00 2024-03-25T19:43:50.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.985+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f6a54004460 con 0x7f6a580b1730 2024-03-25T19:43:50.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.985+0000 7f6a4e7fc640 10 client.14372.objecter ms_dispatch 0x7f6a58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:50.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.985+0000 7f6a4e7fc640 3 client.14372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:50.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.985+0000 7f6a4e7fc640 3 client.14372.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:50.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.985+0000 7f6a4e7fc640 20 client.14372.objecter dump_active .. 0 homeless 2024-03-25T19:43:50.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.985+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a540504f0 con 0x7f6a580b1730 2024-03-25T19:43:50.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.985+0000 7f6a4e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:50.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:50.985+0000 7f6a4e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:51.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.273+0000 7f6a5fdc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:51.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.273+0000 7f6a5fdc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:51.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.273+0000 7f6a5fdc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:43:51.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.273+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a58134d00 con 0x7f6a580b1730 2024-03-25T19:43:51.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.333+0000 7f6a4e7fc640 1 -- 172.21.15.71:0/3571199798 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f6a54052420 con 0x7f6a580b1730 2024-03-25T19:43:51.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.334+0000 7f6a4e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:51.335 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.334+0000 7f6a4e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:51.335 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:43:51.335 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:51.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 10 librados: watch_flush enter 2024-03-25T19:43:51.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 10 librados: watch_flush exit 2024-03-25T19:43:51.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 20 client.14372.objecter shutdown clearing up homeless session... 2024-03-25T19:43:51.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 10 client.14372.objecter successfully canceled tick 2024-03-25T19:43:51.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 >> v1:172.21.15.71:6800/3911735624 conn(0x7f6a3c06dd00 legacy=0x7f6a3c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 10 monclient: shutdown 2024-03-25T19:43:51.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:51.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 >> v1:172.21.15.71:6790/0 conn(0x7f6a580b1730 legacy=0x7f6a58135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:51.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 shutdown_connections 2024-03-25T19:43:51.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 >> 172.21.15.71:0/3571199798 conn(0x7f6a580617a0 msgr2=0x7f6a5813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:51.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 shutdown_connections 2024-03-25T19:43:51.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 1 -- 172.21.15.71:0/3571199798 wait complete. 2024-03-25T19:43:51.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:51.336+0000 7f6a5fdc6640 1 librados: shutdown 2024-03-25T19:43:51.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:56.738 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:39.494 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.742 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:39.499 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.771 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:39.527 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.794 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:39.514 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.804 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:39.524 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.821 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:39.577 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.824 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:39.543 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.830 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:39.586 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.861 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:39.618 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:58.156 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:58.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.270+0000 7f6899d0e640 10 monclient: get_monmap_and_config 2024-03-25T19:43:58.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.270+0000 7f6899d0e640 10 monclient: build_initial_monmap 2024-03-25T19:43:58.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.270+0000 7f6899d0e640 10 monclient: monmap: 2024-03-25T19:43:58.270 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:58.270 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:58.270 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:43:58.271682+0000 2024-03-25T19:43:58.271 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:43:58.271682+0000 2024-03-25T19:43:58.271 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:58.271 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:58.271 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:58.271 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:43:58.271 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:43:58.271 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:58.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.271+0000 7f6899d0e640 1 Processor -- start 2024-03-25T19:43:58.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.271+0000 7f6899d0e640 1 -- start start 2024-03-25T19:43:58.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.271+0000 7f6899d0e640 10 monclient: init 2024-03-25T19:43:58.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f68940b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f68940b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f689414be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68940af940 con 0x7f68940b4d60 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68940b0a70 con 0x7f689414be90 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68940b0d50 con 0x7f68940b1730 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:58.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.272+0000 7f6899d0e640 10 monclient(hunting): authenticate will time out at 5044.584360s 2024-03-25T19:43:58.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.281+0000 7f68937fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f68940b4d60 0x7f689413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43198/0 (socket says 172.21.15.71:43198) 2024-03-25T19:43:58.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.281+0000 7f68937fe640 1 -- 172.21.15.71:0/2667912430 learned_addr learned my addr 172.21.15.71:0/2667912430 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:58.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.292+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818716374 0 0) 0x7f68940af940 con 0x7f68940b4d60 2024-03-25T19:43:58.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.292+0000 7f68927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:58.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.292+0000 7f68927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:58.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.292+0000 7f68927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:58.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.292+0000 7f68927fc640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:43:58.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.292+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f687c003520 con 0x7f68940b4d60 2024-03-25T19:43:58.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.303+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887218289 0 0) 0x7f68940b0a70 con 0x7f689414be90 2024-03-25T19:43:58.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.303+0000 7f68927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:58.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.303+0000 7f68927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:58.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.303+0000 7f68927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:58.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.303+0000 7f68927fc640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:43:58.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.303+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68940af940 con 0x7f689414be90 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 472109505 0 0) 0x7f68940af940 con 0x7f689414be90 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 >> v1:172.21.15.71:6790/0 conn(0x7f68940b1730 legacy=0x7f68940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 >> v1:172.21.15.71:6789/0 conn(0x7f68940b4d60 legacy=0x7f689413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 1 monclient: found mon.noname-c 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68940b1030 con 0x7f689414be90 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:58.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f688c0019f0 con 0x7f689414be90 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f68927fc640 10 monclient: dump: 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:58.308 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:58.309 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:58.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f6899d0e640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:43:58.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.307+0000 7f6899d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.331+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f688c001b50 con 0x7f689414be90 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.331+0000 7f68927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.331+0000 7f68927fc640 1 -- 172.21.15.71:0/2667912430 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f688c001f20 con 0x7f689414be90 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.331+0000 7f6899d0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.331+0000 7f6899d0e640 4 set_mon_vals no callback set 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.331+0000 7f68927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.331+0000 7f68927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.331+0000 7f68927fc640 10 monclient: dump: 2024-03-25T19:43:58.331 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:58.332 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:58.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.332+0000 7f6899d0e640 10 monclient: shutdown 2024-03-25T19:43:58.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.332+0000 7f6899d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:58.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.332+0000 7f6899d0e640 1 -- 172.21.15.71:0/2667912430 >> v1:172.21.15.193:6789/0 conn(0x7f689414be90 legacy=0x7f689414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.335 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.332+0000 7f6899d0e640 1 -- 172.21.15.71:0/2667912430 shutdown_connections 2024-03-25T19:43:58.335 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.332+0000 7f6899d0e640 1 -- 172.21.15.71:0/2667912430 >> 172.21.15.71:0/2667912430 conn(0x7f68940617a0 msgr2=0x7f6894061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.332+0000 7f6899d0e640 1 -- 172.21.15.71:0/2667912430 shutdown_connections 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.332+0000 7f6899d0e640 1 -- 172.21.15.71:0/2667912430 wait complete. 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 10 monclient: build_initial_monmap 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 10 monclient: monmap: 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:58.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 1 librados: starting msgr at 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 1 librados: starting objecter 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 1 Processor -- start 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 1 -- start start 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 1 librados: setting wanted keys 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 1 librados: calling monclient init 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.333+0000 7f6899d0e640 10 monclient: init 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f68940b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f68940b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f689414be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68941422f0 con 0x7f68940b4d60 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689413d490 con 0x7f689414be90 2024-03-25T19:43:58.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6894143290 con 0x7f68940b1730 2024-03-25T19:43:58.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:58.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.334+0000 7f6899d0e640 10 monclient(hunting): authenticate will time out at 5044.646326s 2024-03-25T19:43:58.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.336+0000 7f6892ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f68940b4d60 0x7f6894132d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43226/0 (socket says 172.21.15.71:43226) 2024-03-25T19:43:58.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.336+0000 7f6892ffd640 1 -- 172.21.15.71:0/2725409835 learned_addr learned my addr 172.21.15.71:0/2725409835 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:43:58.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.339+0000 7f687bfff640 10 client.?.objecter ms_handle_connect 0x7f68940b4d60 2024-03-25T19:43:58.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.339+0000 7f687bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:58.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.340+0000 7f687bfff640 10 client.?.objecter ms_handle_connect 0x7f689414be90 2024-03-25T19:43:58.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.340+0000 7f687bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.349+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111666781 0 0) 0x7f68941422f0 con 0x7f68940b4d60 2024-03-25T19:43:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.349+0000 7f687bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.349+0000 7f687bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.349+0000 7f687bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.349+0000 7f687bfff640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:43:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.349+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6880004eb0 con 0x7f68940b4d60 2024-03-25T19:43:58.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.350+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083252417 0 0) 0x7f689413d490 con 0x7f689414be90 2024-03-25T19:43:58.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.350+0000 7f687bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:58.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.350+0000 7f687bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:58.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.350+0000 7f687bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:58.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.350+0000 7f687bfff640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:43:58.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.350+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68941422f0 con 0x7f689414be90 2024-03-25T19:43:58.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.353+0000 7f687bfff640 10 client.?.objecter ms_handle_connect 0x7f68940b1730 2024-03-25T19:43:58.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.353+0000 7f687bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:58.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.364+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828206200 0 0) 0x7f6894143290 con 0x7f68940b1730 2024-03-25T19:43:58.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.364+0000 7f687bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:58.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.364+0000 7f687bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:58.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.364+0000 7f687bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:58.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.364+0000 7f687bfff640 10 monclient(hunting): my global_id is 14378 2024-03-25T19:43:58.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.364+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f689413d490 con 0x7f68940b1730 2024-03-25T19:43:58.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.366+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3361831417 0 0) 0x7f6880004eb0 con 0x7f68940b4d60 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.366+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6894143290 con 0x7f68940b4d60 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.366+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6888001a30 con 0x7f68940b4d60 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.366+0000 7f687bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.366+0000 7f687bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.366+0000 7f687bfff640 10 monclient(hunting): dump: 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:58.366 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:58.367 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.366+0000 7f687bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.368+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 726134104 0 0) 0x7f68941422f0 con 0x7f689414be90 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.368+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6880004eb0 con 0x7f689414be90 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.368+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f688c001e60 con 0x7f689414be90 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.368+0000 7f687bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.368+0000 7f687bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.368+0000 7f687bfff640 10 monclient(hunting): dump: 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:58.368 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:58.369 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:58.369 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:58.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4171925037 0 0) 0x7f6880004eb0 con 0x7f689414be90 2024-03-25T19:43:58.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 >> v1:172.21.15.71:6790/0 conn(0x7f68940b1730 legacy=0x7f6894141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 >> v1:172.21.15.71:6789/0 conn(0x7f68940b4d60 legacy=0x7f6894132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:58.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 1 monclient: found mon.b 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689413d6e0 con 0x7f689414be90 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 10 monclient: _finish_auth 0 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f687bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 5 monclient: authenticate success, global_id 4375 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 10 monclient: _renew_subs 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6894141100 con 0x7f689414be90 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 10 monclient: _renew_subs 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6894133430 con 0x7f689414be90 2024-03-25T19:43:58.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.377+0000 7f6899d0e640 1 librados: init done 2024-03-25T19:43:58.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.378+0000 7f6899d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:58.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.378+0000 7f6899d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:58.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.378+0000 7f6899d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:58.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.378+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68940627b0 con 0x7f689414be90 2024-03-25T19:43:58.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.647+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f688c002060 con 0x7f689414be90 2024-03-25T19:43:58.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.647+0000 7f687bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.647+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f688c003dc0 con 0x7f689414be90 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.647+0000 7f687bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.647+0000 7f687bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.647+0000 7f687bfff640 10 monclient: dump: 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:43:58.648 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:43:58.649 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:43:58.649 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:43:58.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.647+0000 7f6891ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:58.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.648+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f688c019d90 con 0x7f689414be90 2024-03-25T19:43:58.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.649+0000 7f68937fe640 10 monclient: get_auth_request con 0x7f688006dfb0 auth_method 0 2024-03-25T19:43:58.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.649+0000 7f687bfff640 10 client.4375.objecter ms_handle_connect 0x7f688006dfb0 2024-03-25T19:43:58.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.653+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f688c053900 con 0x7f689414be90 2024-03-25T19:43:58.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.653+0000 7f687bfff640 10 client.4375.objecter ms_dispatch 0x7f6894000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:58.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.653+0000 7f687bfff640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:58.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.653+0000 7f687bfff640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:58.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.653+0000 7f687bfff640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T19:43:58.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.653+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f688c04f240 con 0x7f689414be90 2024-03-25T19:43:58.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.654+0000 7f687bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:58.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.654+0000 7f687bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.883+0000 7f6899d0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.883+0000 7f6899d0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:58.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.883+0000 7f6899d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:43:58.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.883+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f689413d870 con 0x7f689414be90 2024-03-25T19:43:58.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.938+0000 7f687bfff640 1 -- 172.21.15.71:0/2725409835 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f688c051340 con 0x7f689414be90 2024-03-25T19:43:58.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.938+0000 7f687bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:58.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.938+0000 7f687bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:58.938 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:43:58.938 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:58.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 10 librados: watch_flush enter 2024-03-25T19:43:58.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 10 librados: watch_flush exit 2024-03-25T19:43:58.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T19:43:58.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 10 client.4375.objecter successfully canceled tick 2024-03-25T19:43:58.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 >> v1:172.21.15.71:6800/3911735624 conn(0x7f688006dfb0 legacy=0x7f6880070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 10 monclient: shutdown 2024-03-25T19:43:58.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:58.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 >> v1:172.21.15.193:6789/0 conn(0x7f689414be90 legacy=0x7f689413dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 shutdown_connections 2024-03-25T19:43:58.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 >> 172.21.15.71:0/2725409835 conn(0x7f68940617a0 msgr2=0x7f68940b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:58.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 shutdown_connections 2024-03-25T19:43:58.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 1 -- 172.21.15.71:0/2725409835 wait complete. 2024-03-25T19:43:58.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:43:58.941+0000 7f6899d0e640 1 librados: shutdown 2024-03-25T19:43:58.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:05.760 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:05.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.867+0000 7fa7624e6640 10 monclient: get_monmap_and_config 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.867+0000 7fa7624e6640 10 monclient: build_initial_monmap 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.867+0000 7fa7624e6640 10 monclient: monmap: 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:44:05.868220+0000 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:44:05.868220+0000 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:44:05.867 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:44:05.868 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 1 Processor -- start 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 1 -- start start 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 10 monclient: init 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 10 monclient(hunting): picked mon.noname-c con 0x7fa75c0b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 10 monclient(hunting): picked mon.noname-a con 0x7fa75c0b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 10 monclient(hunting): picked mon.noname-b con 0x7fa75c14be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:05.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c0af940 con 0x7fa75c0b1730 2024-03-25T19:44:05.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c0b0a70 con 0x7fa75c0b4d60 2024-03-25T19:44:05.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c0b0d50 con 0x7fa75c14be90 2024-03-25T19:44:05.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:05.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.868+0000 7fa7624e6640 10 monclient(hunting): authenticate will time out at 5052.180529s 2024-03-25T19:44:05.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.869+0000 7fa75bfff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa75c0b4d60 0x7fa75c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:58160/0 (socket says 172.21.15.71:58160) 2024-03-25T19:44:05.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.869+0000 7fa75bfff640 1 -- 172.21.15.71:0/2971366415 learned_addr learned my addr 172.21.15.71:0/2971366415 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:05.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.904+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132435657 0 0) 0x7fa75c0b0d50 con 0x7fa75c14be90 2024-03-25T19:44:05.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.904+0000 7fa75affd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.904+0000 7fa75affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.904+0000 7fa75affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.904+0000 7fa75affd640 10 monclient(hunting): my global_id is 14381 2024-03-25T19:44:05.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.904+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa744003520 con 0x7fa75c14be90 2024-03-25T19:44:05.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263464144 0 0) 0x7fa744003520 con 0x7fa75c14be90 2024-03-25T19:44:05.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 >> v1:172.21.15.193:6789/0 conn(0x7fa75c0b4d60 legacy=0x7fa75c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 >> v1:172.21.15.71:6789/0 conn(0x7fa75c0b1730 legacy=0x7fa75c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 1 monclient: found mon.noname-b 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c0b1030 con 0x7fa75c14be90 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 10 monclient: _finish_auth 0 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa754001a10 con 0x7fa75c14be90 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa7624e6640 5 monclient: authenticate success, global_id 14381 2024-03-25T19:44:05.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa7624e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa75affd640 10 monclient: dump: 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:05.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.912+0000 7fa7624e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.942+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa754001b70 con 0x7fa75c14be90 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.942+0000 7fa75affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.942+0000 7fa75affd640 1 -- 172.21.15.71:0/2971366415 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa754001f80 con 0x7fa75c14be90 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.942+0000 7fa75affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.942+0000 7fa75affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.942+0000 7fa75affd640 10 monclient: dump: 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:05.942 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:05.943 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:05.943 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:05.943 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:05.943 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:05.943 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:05.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.942+0000 7fa7624e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:05.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.942+0000 7fa7624e6640 4 set_mon_vals no callback set 2024-03-25T19:44:05.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.943+0000 7fa7624e6640 10 monclient: shutdown 2024-03-25T19:44:05.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.943+0000 7fa7624e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:05.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.943+0000 7fa7624e6640 1 -- 172.21.15.71:0/2971366415 >> v1:172.21.15.71:6790/0 conn(0x7fa75c14be90 legacy=0x7fa75c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.943+0000 7fa7624e6640 1 -- 172.21.15.71:0/2971366415 shutdown_connections 2024-03-25T19:44:05.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.943+0000 7fa7624e6640 1 -- 172.21.15.71:0/2971366415 >> 172.21.15.71:0/2971366415 conn(0x7fa75c0617a0 msgr2=0x7fa75c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:05.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.943+0000 7fa7624e6640 1 -- 172.21.15.71:0/2971366415 shutdown_connections 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.943+0000 7fa7624e6640 1 -- 172.21.15.71:0/2971366415 wait complete. 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 10 monclient: build_initial_monmap 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 10 monclient: monmap: 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:44:05.946 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 1 librados: starting msgr at 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 1 librados: starting objecter 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 1 Processor -- start 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 1 -- start start 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 1 librados: setting wanted keys 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 1 librados: calling monclient init 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.944+0000 7fa7624e6640 10 monclient: init 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 10 monclient(hunting): picked mon.noname-b con 0x7fa75c0b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 10 monclient(hunting): picked mon.noname-c con 0x7fa75c0b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 10 monclient(hunting): picked mon.noname-a con 0x7fa75c14be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c137b10 con 0x7fa75c14be90 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c1377e0 con 0x7fa75c0b1730 2024-03-25T19:44:05.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c137fa0 con 0x7fa75c0b4d60 2024-03-25T19:44:05.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:05.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.945+0000 7fa7624e6640 10 monclient(hunting): authenticate will time out at 5052.257645s 2024-03-25T19:44:05.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.963+0000 7fa760a83640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fa75c14be90 0x7fa75c1e3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47868/0 (socket says 172.21.15.71:47868) 2024-03-25T19:44:05.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.963+0000 7fa760a83640 1 -- 172.21.15.71:0/1690287185 learned_addr learned my addr 172.21.15.71:0/1690287185 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:05.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.964+0000 7fa758ff9640 10 client.?.objecter ms_handle_connect 0x7fa75c14be90 2024-03-25T19:44:05.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.964+0000 7fa758ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:05.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.967+0000 7fa758ff9640 10 client.?.objecter ms_handle_connect 0x7fa75c0b1730 2024-03-25T19:44:05.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.967+0000 7fa758ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:05.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.976+0000 7fa758ff9640 10 client.?.objecter ms_handle_connect 0x7fa75c0b4d60 2024-03-25T19:44:05.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.976+0000 7fa758ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:05.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.980+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684244496 0 0) 0x7fa75c137b10 con 0x7fa75c14be90 2024-03-25T19:44:05.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.980+0000 7fa758ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.980+0000 7fa758ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.980+0000 7fa758ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.980+0000 7fa758ff9640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:44:05.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.980+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa748004eb0 con 0x7fa75c14be90 2024-03-25T19:44:05.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.984+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260183776 0 0) 0x7fa75c1377e0 con 0x7fa75c0b1730 2024-03-25T19:44:05.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.984+0000 7fa758ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.984+0000 7fa758ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.984+0000 7fa758ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.984+0000 7fa758ff9640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:44:05.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.984+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa75c137b10 con 0x7fa75c0b1730 2024-03-25T19:44:05.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.988+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 776758644 0 0) 0x7fa748004eb0 con 0x7fa75c14be90 2024-03-25T19:44:05.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.988+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa75c1377e0 con 0x7fa75c14be90 2024-03-25T19:44:05.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.988+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa754001f40 con 0x7fa75c14be90 2024-03-25T19:44:05.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.988+0000 7fa758ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.988+0000 7fa758ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.988+0000 7fa758ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:05.989 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:05.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.988+0000 7fa758ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:05.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.990+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073955619 0 0) 0x7fa75c137fa0 con 0x7fa75c0b4d60 2024-03-25T19:44:05.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.990+0000 7fa758ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.990+0000 7fa758ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.990+0000 7fa758ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.990+0000 7fa758ff9640 10 monclient(hunting): my global_id is 14384 2024-03-25T19:44:05.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.990+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa748004eb0 con 0x7fa75c0b4d60 2024-03-25T19:44:05.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1729142177 0 0) 0x7fa75c1377e0 con 0x7fa75c14be90 2024-03-25T19:44:05.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 >> v1:172.21.15.71:6790/0 conn(0x7fa75c0b4d60 legacy=0x7fa75c1daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 >> v1:172.21.15.193:6789/0 conn(0x7fa75c0b1730 legacy=0x7fa75c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:05.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 1 monclient: found mon.a 2024-03-25T19:44:05.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:05.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c142030 con 0x7fa75c14be90 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa758ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 5 monclient: authenticate success, global_id 4377 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 10 monclient: _renew_subs 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c140840 con 0x7fa75c14be90 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 10 monclient: _renew_subs 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c1357a0 con 0x7fa75c14be90 2024-03-25T19:44:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.998+0000 7fa7624e6640 1 librados: init done 2024-03-25T19:44:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.999+0000 7fa7624e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.999+0000 7fa7624e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.999+0000 7fa7624e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:05.999+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c0627b0 con 0x7fa75c14be90 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.288+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa754003c20 con 0x7fa75c14be90 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.288+0000 7fa758ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.288+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa754003fb0 con 0x7fa75c14be90 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.288+0000 7fa758ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.288+0000 7fa758ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.288+0000 7fa758ff9640 10 monclient: dump: 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:06.288 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:06.289 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:06.289 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:06.289 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:06.289 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:06.289 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:06.289 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:06.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.288+0000 7fa75a7fc640 4 set_mon_vals no callback set 2024-03-25T19:44:06.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.289+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fa7540041d0 con 0x7fa75c14be90 2024-03-25T19:44:06.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.289+0000 7fa75bfff640 10 monclient: get_auth_request con 0x7fa74806de20 auth_method 0 2024-03-25T19:44:06.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.289+0000 7fa758ff9640 10 client.4377.objecter ms_handle_connect 0x7fa74806de20 2024-03-25T19:44:06.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.295+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fa754054ab0 con 0x7fa75c14be90 2024-03-25T19:44:06.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.295+0000 7fa758ff9640 10 client.4377.objecter ms_dispatch 0x7fa75c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:06.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.295+0000 7fa758ff9640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:06.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.295+0000 7fa758ff9640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:06.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.295+0000 7fa758ff9640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-25T19:44:06.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.295+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7540505c0 con 0x7fa75c14be90 2024-03-25T19:44:06.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.295+0000 7fa758ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:06.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.295+0000 7fa758ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:06.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.551+0000 7fa7624e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:06.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.552+0000 7fa7624e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:06.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.552+0000 7fa7624e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:06.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.552+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa75c137cc0 con 0x7fa75c14be90 2024-03-25T19:44:06.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.575+0000 7fa758ff9640 1 -- 172.21.15.71:0/1690287185 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fa7540524f0 con 0x7fa75c14be90 2024-03-25T19:44:06.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.575+0000 7fa758ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:06.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.575+0000 7fa758ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:06.575 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:44:06.575 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:06.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 10 librados: watch_flush enter 2024-03-25T19:44:06.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 10 librados: watch_flush exit 2024-03-25T19:44:06.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-25T19:44:06.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 10 client.4377.objecter successfully canceled tick 2024-03-25T19:44:06.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 >> v1:172.21.15.71:6800/3911735624 conn(0x7fa74806de20 legacy=0x7fa7480702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:06.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 10 monclient: shutdown 2024-03-25T19:44:06.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:06.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 >> v1:172.21.15.71:6789/0 conn(0x7fa75c14be90 legacy=0x7fa75c1e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:06.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 shutdown_connections 2024-03-25T19:44:06.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 >> 172.21.15.71:0/1690287185 conn(0x7fa75c0617a0 msgr2=0x7fa75c13c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:06.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 shutdown_connections 2024-03-25T19:44:06.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 1 -- 172.21.15.71:0/1690287185 wait complete. 2024-03-25T19:44:06.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:06.578+0000 7fa7624e6640 1 librados: shutdown 2024-03-25T19:44:06.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:13.396 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.514+0000 7f7ffe977640 10 monclient: get_monmap_and_config 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.514+0000 7f7ffe977640 10 monclient: build_initial_monmap 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.514+0000 7f7ffe977640 10 monclient: monmap: 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:44:13.515410+0000 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:44:13.515410+0000 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:13.514 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:13.515 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:44:13.515 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:44:13.515 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.515+0000 7f7ffe977640 1 Processor -- start 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.515+0000 7f7ffe977640 1 -- start start 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.515+0000 7f7ffe977640 10 monclient: init 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff80a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff81430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8142520 con 0x7f7ff813f710 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80b28a0 con 0x7f7ff80a98f0 2024-03-25T19:44:13.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80b2a30 con 0x7f7ff81430c0 2024-03-25T19:44:13.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:13.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.516+0000 7f7ffe977640 10 monclient(hunting): authenticate will time out at 5059.828209s 2024-03-25T19:44:13.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.520+0000 7f7ff7fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7ff80a98f0 0x7f7ff80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:58104/0 (socket says 172.21.15.71:58104) 2024-03-25T19:44:13.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.520+0000 7f7ff7fff640 1 -- 172.21.15.71:0/3714457858 learned_addr learned my addr 172.21.15.71:0/3714457858 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:13.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.526+0000 7f7ff7fff640 1 -- 172.21.15.71:0/3714457858 >> v1:172.21.15.193:6789/0 conn(0x7f7ff80a98f0 legacy=0x7f7ff80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T19:44:13.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.526+0000 7f7ff7fff640 1 -- 172.21.15.71:0/3714457858 >> v1:172.21.15.193:6789/0 conn(0x7f7ff80a98f0 legacy=0x7f7ff80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:44:13.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.526+0000 7f7ff7fff640 1 --1- 172.21.15.71:0/3714457858 >> v1:172.21.15.193:6789/0 conn(0x7f7ff80a98f0 0x7f7ff80a9cd0 :-1 s=OPENED pgs=23126 cs=1 l=1).handle_message read tag failed 2024-03-25T19:44:13.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.526+0000 7f7ff7fff640 1 --1- 172.21.15.71:0/3714457858 >> v1:172.21.15.193:6789/0 conn(0x7f7ff80a98f0 0x7f7ff80a9cd0 :-1 s=OPENED pgs=23126 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:44:13.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.526+0000 7f7ff6ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.193:6789/0 2024-03-25T19:44:13.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.559+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30398447 0 0) 0x7f7ff80b2a30 con 0x7f7ff81430c0 2024-03-25T19:44:13.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.560+0000 7f7ff6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.560+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.560+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.560+0000 7f7ff6ffd640 10 monclient(hunting): my global_id is 14387 2024-03-25T19:44:13.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.560+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0003520 con 0x7f7ff81430c0 2024-03-25T19:44:13.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.562+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370694605 0 0) 0x7f7ff8142520 con 0x7f7ff813f710 2024-03-25T19:44:13.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.562+0000 7f7ff6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.562+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.562+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.562+0000 7f7ff6ffd640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:44:13.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.562+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff80b2a30 con 0x7f7ff813f710 2024-03-25T19:44:13.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2367203447 0 0) 0x7f7ff80b2a30 con 0x7f7ff813f710 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 >> v1:172.21.15.71:6790/0 conn(0x7f7ff81430c0 legacy=0x7f7ff814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 >> v1:172.21.15.193:6789/0 conn(0x7f7ff80a98f0 legacy=0x7f7ff80a9cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff80b44c0 con 0x7f7ff813f710 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ff6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ffe977640 5 monclient: authenticate success, global_id 4380 2024-03-25T19:44:13.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.568+0000 7f7ffe977640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:13.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7fec0015f0 con 0x7f7ff813f710 2024-03-25T19:44:13.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:13.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 10 monclient: dump: 2024-03-25T19:44:13.585 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:13.585 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ffe977640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fec001880 con 0x7f7ff813f710 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:13.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ffe977640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 1 -- 172.21.15.71:0/3714457858 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7fec001c10 con 0x7f7ff813f710 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ffe977640 4 set_mon_vals no callback set 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.585+0000 7f7ff6ffd640 10 monclient: dump: 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:13.587 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:13.588 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:13.588 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.586+0000 7f7ffe977640 10 monclient: shutdown 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.586+0000 7f7ffe977640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.586+0000 7f7ffe977640 1 -- 172.21.15.71:0/3714457858 >> v1:172.21.15.71:6789/0 conn(0x7f7ff813f710 legacy=0x7f7ff81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.586+0000 7f7ffe977640 1 -- 172.21.15.71:0/3714457858 shutdown_connections 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.586+0000 7f7ffe977640 1 -- 172.21.15.71:0/3714457858 >> 172.21.15.71:0/3714457858 conn(0x7f7ff80b7ea0 msgr2=0x7f7ff80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.586+0000 7f7ffe977640 1 -- 172.21.15.71:0/3714457858 shutdown_connections 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.586+0000 7f7ffe977640 1 -- 172.21.15.71:0/3714457858 wait complete. 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.587+0000 7f7ffe977640 10 monclient: build_initial_monmap 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.587+0000 7f7ffe977640 10 monclient: monmap: 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:13.590 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.587+0000 7f7ffe977640 1 librados: starting msgr at 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.587+0000 7f7ffe977640 1 librados: starting objecter 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.587+0000 7f7ffe977640 1 Processor -- start 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 1 -- start start 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 1 librados: setting wanted keys 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 1 librados: calling monclient init 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 10 monclient: init 2024-03-25T19:44:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff80a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff813f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff81430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff81d7670 con 0x7f7ff80a98f0 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff81d7950 con 0x7f7ff813f710 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff81d84f0 con 0x7f7ff81430c0 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.588+0000 7f7ffe977640 10 monclient(hunting): authenticate will time out at 5059.900820s 2024-03-25T19:44:13.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.604+0000 7f7ff77fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7ff813f710 0x7f7ff8138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:58168/0 (socket says 172.21.15.71:58168) 2024-03-25T19:44:13.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.604+0000 7f7ff77fe640 1 -- 172.21.15.71:0/611443841 learned_addr learned my addr 172.21.15.71:0/611443841 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.604+0000 7f7ff4ff9640 10 client.?.objecter ms_handle_connect 0x7f7ff813f710 2024-03-25T19:44:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.605+0000 7f7ff4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.612+0000 7f7ff4ff9640 10 client.?.objecter ms_handle_connect 0x7f7ff81430c0 2024-03-25T19:44:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.612+0000 7f7ff4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.612+0000 7f7ff4ff9640 10 client.?.objecter ms_handle_connect 0x7f7ff80a98f0 2024-03-25T19:44:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.612+0000 7f7ff4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:13.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.618+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976805505 0 0) 0x7f7ff81d7950 con 0x7f7ff813f710 2024-03-25T19:44:13.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.618+0000 7f7ff4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.618+0000 7f7ff4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.618+0000 7f7ff4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.618+0000 7f7ff4ff9640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:44:13.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.618+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff813f710 2024-03-25T19:44:13.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.624+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984252541 0 0) 0x7f7ff81d7670 con 0x7f7ff80a98f0 2024-03-25T19:44:13.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.624+0000 7f7ff4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.624+0000 7f7ff4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.624+0000 7f7ff4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.624+0000 7f7ff4ff9640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:44:13.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.625+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff81d7950 con 0x7f7ff80a98f0 2024-03-25T19:44:13.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.625+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388088329 0 0) 0x7f7ff81d84f0 con 0x7f7ff81430c0 2024-03-25T19:44:13.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.625+0000 7f7ff4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.625+0000 7f7ff4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.625+0000 7f7ff4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.625+0000 7f7ff4ff9640 10 monclient(hunting): my global_id is 14390 2024-03-25T19:44:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.625+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff81d7670 con 0x7f7ff81430c0 2024-03-25T19:44:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.631+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2255486437 0 0) 0x7f7fe4004eb0 con 0x7f7ff813f710 2024-03-25T19:44:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.631+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff81d84f0 con 0x7f7ff813f710 2024-03-25T19:44:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.631+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7fec001c50 con 0x7f7ff813f710 2024-03-25T19:44:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.631+0000 7f7ff4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.631+0000 7f7ff4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.631+0000 7f7ff4ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:13.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.631+0000 7f7ff4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:13.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.640+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086974331 0 0) 0x7f7ff81d7950 con 0x7f7ff80a98f0 2024-03-25T19:44:13.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.640+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff80a98f0 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.640+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7fe80018b0 con 0x7f7ff80a98f0 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.640+0000 7f7ff4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.640+0000 7f7ff4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.640+0000 7f7ff4ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:13.641 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1585540708 0 0) 0x7f7ff81d84f0 con 0x7f7ff813f710 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 >> v1:172.21.15.71:6790/0 conn(0x7f7ff81430c0 legacy=0x7f7ff81e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 >> v1:172.21.15.71:6789/0 conn(0x7f7ff80a98f0 legacy=0x7f7ff81d7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 1 monclient: found mon.b 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff80abb50 con 0x7f7ff813f710 2024-03-25T19:44:13.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ff4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 5 monclient: authenticate success, global_id 4384 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 10 monclient: _renew_subs 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff81d51f0 con 0x7f7ff813f710 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 10 monclient: _renew_subs 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff8138a50 con 0x7f7ff813f710 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.645+0000 7f7ffe977640 1 librados: init done 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.646+0000 7f7ffe977640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.646+0000 7f7ffe977640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.646+0000 7f7ffe977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.646+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff814f2d0 con 0x7f7ff813f710 2024-03-25T19:44:13.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.924+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fec003960 con 0x7f7ff813f710 2024-03-25T19:44:13.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.924+0000 7f7ff4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:13.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.924+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7fec003d70 con 0x7f7ff813f710 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.924+0000 7f7ff4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.924+0000 7f7ff4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.924+0000 7f7ff4ff9640 10 monclient: dump: 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:13.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.924+0000 7f7ff67fc640 4 set_mon_vals no callback set 2024-03-25T19:44:13.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.925+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f7fec003f90 con 0x7f7ff813f710 2024-03-25T19:44:13.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.926+0000 7f7ff7fff640 10 monclient: get_auth_request con 0x7f7fe406dec0 auth_method 0 2024-03-25T19:44:13.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.926+0000 7f7ff4ff9640 10 client.4384.objecter ms_handle_connect 0x7f7fe406dec0 2024-03-25T19:44:13.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.930+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f7fec054910 con 0x7f7ff813f710 2024-03-25T19:44:13.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.930+0000 7f7ff4ff9640 10 client.4384.objecter ms_dispatch 0x7f7ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:13.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.930+0000 7f7ff4ff9640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:13.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.930+0000 7f7ff4ff9640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:13.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.931+0000 7f7ff4ff9640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T19:44:13.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.931+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fec050420 con 0x7f7ff813f710 2024-03-25T19:44:13.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.931+0000 7f7ff4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:13.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:13.931+0000 7f7ff4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:14.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.172+0000 7f7ffe977640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:14.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.172+0000 7f7ffe977640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:14.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.172+0000 7f7ffe977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:14.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.172+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ff80627b0 con 0x7f7ff813f710 2024-03-25T19:44:14.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.198+0000 7f7ff4ff9640 1 -- 172.21.15.71:0/611443841 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f7fec052350 con 0x7f7ff813f710 2024-03-25T19:44:14.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.198+0000 7f7ff4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:14.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.198+0000 7f7ff4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:14.198 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:44:14.199 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 10 librados: watch_flush enter 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 10 librados: watch_flush exit 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 10 client.4384.objecter successfully canceled tick 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 >> v1:172.21.15.71:6800/3911735624 conn(0x7f7fe406dec0 legacy=0x7f7fe4070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 10 monclient: shutdown 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 >> v1:172.21.15.193:6789/0 conn(0x7f7ff813f710 legacy=0x7f7ff8138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:14.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 shutdown_connections 2024-03-25T19:44:14.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 >> 172.21.15.71:0/611443841 conn(0x7f7ff80b7ea0 msgr2=0x7f7ff8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:14.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 shutdown_connections 2024-03-25T19:44:14.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 1 -- 172.21.15.71:0/611443841 wait complete. 2024-03-25T19:44:14.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:14.201+0000 7f7ffe977640 1 librados: shutdown 2024-03-25T19:44:14.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:16.848 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:59.604 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.855 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:59.611 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.901 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:03:59.657 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.905 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:59.625 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.909 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:59.629 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.940 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:59.660 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.960 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:59.716 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.966 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:59.722 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.997 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:03:59.753 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:21.019 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:21.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.135+0000 7f07e4c96640 10 monclient: get_monmap_and_config 2024-03-25T19:44:21.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.135+0000 7f07e4c96640 10 monclient: build_initial_monmap 2024-03-25T19:44:21.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.135+0000 7f07e4c96640 10 monclient: monmap: 2024-03-25T19:44:21.135 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:21.135 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:21.135 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:44:21.136443+0000 2024-03-25T19:44:21.135 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:44:21.136443+0000 2024-03-25T19:44:21.136 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:21.136 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:21.136 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:21.136 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:44:21.136 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:44:21.136 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 1 Processor -- start 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 1 -- start start 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 10 monclient: init 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 10 monclient(hunting): picked mon.noname-c con 0x7f07e00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 10 monclient(hunting): picked mon.noname-b con 0x7f07e013f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 10 monclient(hunting): picked mon.noname-a con 0x7f07e01430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e0142520 con 0x7f07e01430c0 2024-03-25T19:44:21.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e00b28a0 con 0x7f07e00a98f0 2024-03-25T19:44:21.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e00b2a30 con 0x7f07e013f710 2024-03-25T19:44:21.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:21.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.136+0000 7f07e4c96640 10 monclient(hunting): authenticate will time out at 5067.448664s 2024-03-25T19:44:21.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.146+0000 7f07d5d9b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f07e013f710 0x7f07e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53766/0 (socket says 172.21.15.71:53766) 2024-03-25T19:44:21.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.146+0000 7f07d5d9b640 1 -- 172.21.15.71:0/631545900 learned_addr learned my addr 172.21.15.71:0/631545900 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:21.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.161+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755289054 0 0) 0x7f07e00b2a30 con 0x7f07e013f710 2024-03-25T19:44:21.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.161+0000 7f07ddd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:21.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.161+0000 7f07ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:21.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.161+0000 7f07ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.161+0000 7f07ddd9b640 10 monclient(hunting): my global_id is 14393 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.161+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07cc003520 con 0x7f07e013f710 2024-03-25T19:44:21.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.167+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687496120 0 0) 0x7f07e00b28a0 con 0x7f07e00a98f0 2024-03-25T19:44:21.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.167+0000 7f07ddd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:21.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.167+0000 7f07ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:21.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.167+0000 7f07ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:21.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.167+0000 7f07ddd9b640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:44:21.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.167+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e00b2a30 con 0x7f07e00a98f0 2024-03-25T19:44:21.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3506425157 0 0) 0x7f07cc003520 con 0x7f07e013f710 2024-03-25T19:44:21.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 >> v1:172.21.15.193:6789/0 conn(0x7f07e00a98f0 legacy=0x7f07e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 >> v1:172.21.15.71:6789/0 conn(0x7f07e01430c0 legacy=0x7f07e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 1 monclient: found mon.noname-b 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e00b44c0 con 0x7f07e013f710 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 10 monclient: _finish_auth 0 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07c8001690 con 0x7f07e013f710 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07e4c96640 5 monclient: authenticate success, global_id 14393 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07ddd9b640 10 monclient: dump: 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.176 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.177 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:21.177 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:21.177 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:21.177 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:21.177 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:21.177 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:21.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.175+0000 7f07e4c96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:21.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.194+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07c80017f0 con 0x7f07e013f710 2024-03-25T19:44:21.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.194+0000 7f07ddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:21.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.194+0000 7f07ddd9b640 1 -- 172.21.15.71:0/631545900 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07c8001c00 con 0x7f07e013f710 2024-03-25T19:44:21.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.194+0000 7f07e4c96640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:21.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.194+0000 7f07e4c96640 4 set_mon_vals no callback set 2024-03-25T19:44:21.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.194+0000 7f07ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:21.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.194+0000 7f07ddd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:21.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.194+0000 7f07ddd9b640 10 monclient: dump: 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:21.195 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.195+0000 7f07e4c96640 10 monclient: shutdown 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.195+0000 7f07e4c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.195+0000 7f07e4c96640 1 -- 172.21.15.71:0/631545900 >> v1:172.21.15.71:6790/0 conn(0x7f07e013f710 legacy=0x7f07e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.195+0000 7f07e4c96640 1 -- 172.21.15.71:0/631545900 shutdown_connections 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.195+0000 7f07e4c96640 1 -- 172.21.15.71:0/631545900 >> 172.21.15.71:0/631545900 conn(0x7f07e00b7ea0 msgr2=0x7f07e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.195+0000 7f07e4c96640 1 -- 172.21.15.71:0/631545900 shutdown_connections 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.195+0000 7f07e4c96640 1 -- 172.21.15.71:0/631545900 wait complete. 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.196+0000 7f07e4c96640 10 monclient: build_initial_monmap 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.196+0000 7f07e4c96640 10 monclient: monmap: 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:44:21.197 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.197+0000 7f07e4c96640 1 librados: starting msgr at 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.197+0000 7f07e4c96640 1 librados: starting objecter 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.197+0000 7f07e4c96640 1 Processor -- start 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.197+0000 7f07e4c96640 1 -- start start 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.197+0000 7f07e4c96640 1 librados: setting wanted keys 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.197+0000 7f07e4c96640 1 librados: calling monclient init 2024-03-25T19:44:21.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.197+0000 7f07e4c96640 10 monclient: init 2024-03-25T19:44:21.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:21.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:21.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 10 monclient(hunting): picked mon.noname-a con 0x7f07e00a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:21.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 10 monclient(hunting): picked mon.noname-b con 0x7f07e013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:21.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 10 monclient(hunting): picked mon.noname-c con 0x7f07e01430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:21.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e013be70 con 0x7f07e00a98f0 2024-03-25T19:44:21.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e0131660 con 0x7f07e013f710 2024-03-25T19:44:21.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e01317f0 con 0x7f07e01430c0 2024-03-25T19:44:21.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:21.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.198+0000 7f07e4c96640 10 monclient(hunting): authenticate will time out at 5067.510394s 2024-03-25T19:44:21.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.204+0000 7f07ded9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f07e01430c0 0x7f07e01e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53790/0 (socket says 172.21.15.71:53790) 2024-03-25T19:44:21.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.204+0000 7f07ded9d640 1 -- 172.21.15.71:0/3734992888 learned_addr learned my addr 172.21.15.71:0/3734992888 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:21.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.208+0000 7f07d77fe640 10 client.?.objecter ms_handle_connect 0x7f07e01430c0 2024-03-25T19:44:21.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.208+0000 7f07d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:21.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.213+0000 7f07d77fe640 10 client.?.objecter ms_handle_connect 0x7f07e00a98f0 2024-03-25T19:44:21.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.213+0000 7f07d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:21.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.214+0000 7f07d77fe640 10 client.?.objecter ms_handle_connect 0x7f07e013f710 2024-03-25T19:44:21.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.214+0000 7f07d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:21.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.220+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179764512 0 0) 0x7f07e0131660 con 0x7f07e013f710 2024-03-25T19:44:21.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.220+0000 7f07d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:21.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.220+0000 7f07d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:21.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.220+0000 7f07d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:21.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.220+0000 7f07d77fe640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:44:21.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.220+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07c0004eb0 con 0x7f07e013f710 2024-03-25T19:44:21.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.222+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870359927 0 0) 0x7f07e01317f0 con 0x7f07e01430c0 2024-03-25T19:44:21.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.222+0000 7f07d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:21.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.222+0000 7f07d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:21.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.222+0000 7f07d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:21.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.222+0000 7f07d77fe640 10 monclient(hunting): my global_id is 14396 2024-03-25T19:44:21.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.222+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e0131660 con 0x7f07e01430c0 2024-03-25T19:44:21.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.223+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129291580 0 0) 0x7f07e013be70 con 0x7f07e00a98f0 2024-03-25T19:44:21.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.223+0000 7f07d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:21.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.223+0000 7f07d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:21.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.223+0000 7f07d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:21.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.223+0000 7f07d77fe640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:44:21.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.223+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e01317f0 con 0x7f07e00a98f0 2024-03-25T19:44:21.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.229+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931708125 0 0) 0x7f07c0004eb0 con 0x7f07e013f710 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.229+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07e013be70 con 0x7f07e013f710 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.229+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07c8001bc0 con 0x7f07e013f710 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.229+0000 7f07d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.229+0000 7f07d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.229+0000 7f07d77fe640 10 monclient(hunting): dump: 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:21.229 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:21.230 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:21.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.229+0000 7f07d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:21.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.232+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715336145 0 0) 0x7f07e0131660 con 0x7f07e01430c0 2024-03-25T19:44:21.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.233+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07c0004eb0 con 0x7f07e01430c0 2024-03-25T19:44:21.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.233+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07d8001bd0 con 0x7f07e01430c0 2024-03-25T19:44:21.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.233+0000 7f07d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:21.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.233+0000 7f07d77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.233+0000 7f07d77fe640 10 monclient(hunting): dump: 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:21.233 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 378433289 0 0) 0x7f07e013be70 con 0x7f07e013f710 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 >> v1:172.21.15.71:6790/0 conn(0x7f07e01430c0 legacy=0x7f07e01e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 >> v1:172.21.15.71:6789/0 conn(0x7f07e00a98f0 legacy=0x7f07e0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 1 monclient: found mon.b 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e0131980 con 0x7f07e013f710 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:21.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.234+0000 7f07d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 10 monclient: _renew_subs 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e0135c60 con 0x7f07e013f710 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 10 monclient: _renew_subs 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e00b7770 con 0x7f07e013f710 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.235+0000 7f07e4c96640 1 librados: init done 2024-03-25T19:44:21.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.236+0000 7f07e4c96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:21.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.236+0000 7f07e4c96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:21.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.236+0000 7f07e4c96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:21.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.236+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e01371d0 con 0x7f07e013f710 2024-03-25T19:44:21.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.452+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07c8004a90 con 0x7f07e013f710 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.452+0000 7f07d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.452+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07c8004da0 con 0x7f07e013f710 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.452+0000 7f07d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.452+0000 7f07d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.452+0000 7f07d77fe640 10 monclient: dump: 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:21.452 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:21.453 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:21.453 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:21.453 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:21.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.452+0000 7f07dd59a640 4 set_mon_vals no callback set 2024-03-25T19:44:21.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.514+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f07c801ad70 con 0x7f07e013f710 2024-03-25T19:44:21.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.515+0000 7f07de59c640 10 monclient: get_auth_request con 0x7f07c006e000 auth_method 0 2024-03-25T19:44:21.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.515+0000 7f07d77fe640 10 client.4390.objecter ms_handle_connect 0x7f07c006e000 2024-03-25T19:44:21.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.518+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f07c8054900 con 0x7f07e013f710 2024-03-25T19:44:21.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.518+0000 7f07d77fe640 10 client.4390.objecter ms_dispatch 0x7f07e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:21.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.518+0000 7f07d77fe640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:21.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.518+0000 7f07d77fe640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:21.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.518+0000 7f07d77fe640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T19:44:21.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.520+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07c8050410 con 0x7f07e013f710 2024-03-25T19:44:21.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.521+0000 7f07d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:21.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.521+0000 7f07d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:21.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.778+0000 7f07e4c96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:21.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.778+0000 7f07e4c96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:21.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.778+0000 7f07e4c96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:21.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.778+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07e0074cc0 con 0x7f07e013f710 2024-03-25T19:44:21.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.857+0000 7f07d77fe640 1 -- 172.21.15.71:0/3734992888 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f07c8052340 con 0x7f07e013f710 2024-03-25T19:44:21.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.857+0000 7f07d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:21.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.857+0000 7f07d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:21.857 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:44:21.857 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:21.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 10 librados: watch_flush enter 2024-03-25T19:44:21.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 10 librados: watch_flush exit 2024-03-25T19:44:21.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 10 client.4390.objecter successfully canceled tick 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 >> v1:172.21.15.71:6800/3911735624 conn(0x7f07c006e000 legacy=0x7f07c00704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 10 monclient: shutdown 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 >> v1:172.21.15.193:6789/0 conn(0x7f07e013f710 legacy=0x7f07e01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 shutdown_connections 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 >> 172.21.15.71:0/3734992888 conn(0x7f07e00b7ea0 msgr2=0x7f07e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 shutdown_connections 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 1 -- 172.21.15.71:0/3734992888 wait complete. 2024-03-25T19:44:21.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:21.859+0000 7f07e4c96640 1 librados: shutdown 2024-03-25T19:44:21.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:28.677 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:28.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.793+0000 7f3d3f49a640 10 monclient: get_monmap_and_config 2024-03-25T19:44:28.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.793+0000 7f3d3f49a640 10 monclient: build_initial_monmap 2024-03-25T19:44:28.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.793+0000 7f3d3f49a640 10 monclient: monmap: 2024-03-25T19:44:28.793 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:44:28.794728+0000 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:44:28.794728+0000 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:44:28.794 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.794+0000 7f3d3f49a640 1 Processor -- start 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.794+0000 7f3d3f49a640 1 -- start start 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.794+0000 7f3d3f49a640 10 monclient: init 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 10 monclient(hunting): picked mon.noname-c con 0x7f3d380b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 10 monclient(hunting): picked mon.noname-b con 0x7f3d380b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3814be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:28.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380af940 con 0x7f3d3814be90 2024-03-25T19:44:28.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380b0a70 con 0x7f3d380b4d60 2024-03-25T19:44:28.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380b0d50 con 0x7f3d380b1730 2024-03-25T19:44:28.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:28.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3f49a640 10 monclient(hunting): authenticate will time out at 5075.107098s 2024-03-25T19:44:28.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3ca35640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3d380b1730 0x7f3d380b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35256/0 (socket says 172.21.15.71:35256) 2024-03-25T19:44:28.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.795+0000 7f3d3ca35640 1 -- 172.21.15.71:0/3014290385 learned_addr learned my addr 172.21.15.71:0/3014290385 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:28.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.832+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726585153 0 0) 0x7f3d380af940 con 0x7f3d3814be90 2024-03-25T19:44:28.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.832+0000 7f3d2bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:28.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.832+0000 7f3d2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:28.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.832+0000 7f3d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:28.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.832+0000 7f3d2bfff640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:44:28.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.832+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d24003520 con 0x7f3d3814be90 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2467416779 0 0) 0x7f3d24003520 con 0x7f3d3814be90 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 >> v1:172.21.15.71:6790/0 conn(0x7f3d380b1730 legacy=0x7f3d380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 >> v1:172.21.15.193:6789/0 conn(0x7f3d380b4d60 legacy=0x7f3d3813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 1 monclient: found mon.noname-a 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380b1030 con 0x7f3d3814be90 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 10 monclient: _finish_auth 0 2024-03-25T19:44:28.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3d30001a10 con 0x7f3d3814be90 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d3f49a640 5 monclient: authenticate success, global_id 4392 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d3f49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d2bfff640 10 monclient: dump: 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:28.862 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:28.863 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:28.863 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:28.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.861+0000 7f3d3f49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:28.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.881+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d30001b70 con 0x7f3d3814be90 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.881+0000 7f3d2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.881+0000 7f3d2bfff640 1 -- 172.21.15.71:0/3014290385 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3d30001f80 con 0x7f3d3814be90 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.881+0000 7f3d3f49a640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.881+0000 7f3d3f49a640 4 set_mon_vals no callback set 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.881+0000 7f3d2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.881+0000 7f3d2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.881+0000 7f3d2bfff640 10 monclient: dump: 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:28.881 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:28.882 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:28.882 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:28.882 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:28.882 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:28.882 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:28.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.882+0000 7f3d3f49a640 10 monclient: shutdown 2024-03-25T19:44:28.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.882+0000 7f3d3f49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:28.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.882+0000 7f3d3f49a640 1 -- 172.21.15.71:0/3014290385 >> v1:172.21.15.71:6789/0 conn(0x7f3d3814be90 legacy=0x7f3d3814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.882+0000 7f3d3f49a640 1 -- 172.21.15.71:0/3014290385 shutdown_connections 2024-03-25T19:44:28.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.882+0000 7f3d3f49a640 1 -- 172.21.15.71:0/3014290385 >> 172.21.15.71:0/3014290385 conn(0x7f3d380617a0 msgr2=0x7f3d38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:28.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.882+0000 7f3d3f49a640 1 -- 172.21.15.71:0/3014290385 shutdown_connections 2024-03-25T19:44:28.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.882+0000 7f3d3f49a640 1 -- 172.21.15.71:0/3014290385 wait complete. 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 10 monclient: build_initial_monmap 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 10 monclient: monmap: 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 1 librados: starting msgr at 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 1 librados: starting objecter 2024-03-25T19:44:28.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 1 Processor -- start 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 1 -- start start 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 1 librados: setting wanted keys 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 1 librados: calling monclient init 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.883+0000 7f3d3f49a640 10 monclient: init 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 10 monclient(hunting): picked mon.noname-c con 0x7f3d380b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 10 monclient(hunting): picked mon.noname-b con 0x7f3d380b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3814be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d38138150 con 0x7f3d3814be90 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d38137c70 con 0x7f3d380b4d60 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d38137e00 con 0x7f3d380b1730 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:28.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.884+0000 7f3d3f49a640 10 monclient(hunting): authenticate will time out at 5075.196338s 2024-03-25T19:44:28.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.892+0000 7f3d3d236640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3d380b1730 0x7f3d38131da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35314/0 (socket says 172.21.15.71:35314) 2024-03-25T19:44:28.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.892+0000 7f3d3d236640 1 -- 172.21.15.71:0/1322144240 learned_addr learned my addr 172.21.15.71:0/1322144240 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:28.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.893+0000 7f3d29ffb640 10 client.?.objecter ms_handle_connect 0x7f3d380b1730 2024-03-25T19:44:28.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.893+0000 7f3d29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:28.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.896+0000 7f3d29ffb640 10 client.?.objecter ms_handle_connect 0x7f3d380b4d60 2024-03-25T19:44:28.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.896+0000 7f3d29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:28.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.901+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222361465 0 0) 0x7f3d38137e00 con 0x7f3d380b1730 2024-03-25T19:44:28.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.901+0000 7f3d29ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:28.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.901+0000 7f3d29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:28.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.901+0000 7f3d29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:28.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.901+0000 7f3d29ffb640 10 monclient(hunting): my global_id is 14402 2024-03-25T19:44:28.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.901+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c004eb0 con 0x7f3d380b1730 2024-03-25T19:44:28.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.906+0000 7f3d29ffb640 10 client.?.objecter ms_handle_connect 0x7f3d3814be90 2024-03-25T19:44:28.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.906+0000 7f3d29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:28.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.912+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867494889 0 0) 0x7f3d38137c70 con 0x7f3d380b4d60 2024-03-25T19:44:28.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.912+0000 7f3d29ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:28.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.912+0000 7f3d29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:28.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.912+0000 7f3d29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:28.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.912+0000 7f3d29ffb640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:44:28.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.912+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d38137e00 con 0x7f3d380b4d60 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.913+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4231537110 0 0) 0x7f3d1c004eb0 con 0x7f3d380b1730 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.913+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d38137c70 con 0x7f3d380b1730 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.913+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3d2c0019d0 con 0x7f3d380b1730 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.913+0000 7f3d29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.913+0000 7f3d29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.913+0000 7f3d29ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:28.913 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:28.914 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:28.914 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:28.914 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:28.914 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:28.914 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:28.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.913+0000 7f3d29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:28.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 954995333 0 0) 0x7f3d38137c70 con 0x7f3d380b1730 2024-03-25T19:44:28.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 >> v1:172.21.15.193:6789/0 conn(0x7f3d380b4d60 legacy=0x7f3d381d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 >> v1:172.21.15.71:6789/0 conn(0x7f3d3814be90 legacy=0x7f3d381d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:28.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 1 monclient: found mon.c 2024-03-25T19:44:28.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:28.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d381386a0 con 0x7f3d380b1730 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 5 monclient: authenticate success, global_id 14402 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 10 monclient: _renew_subs 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d38144e00 con 0x7f3d380b1730 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 10 client.14402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 10 monclient: _renew_subs 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d38132690 con 0x7f3d380b1730 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.919+0000 7f3d3f49a640 1 librados: init done 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.920+0000 7f3d3f49a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.920+0000 7f3d3f49a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.920+0000 7f3d3f49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:28.920+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d381451a0 con 0x7f3d380b1730 2024-03-25T19:44:29.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.220+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d2c0035f0 con 0x7f3d380b1730 2024-03-25T19:44:29.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.220+0000 7f3d29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:29.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.220+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3d2c003980 con 0x7f3d380b1730 2024-03-25T19:44:29.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.220+0000 7f3d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:29.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.220+0000 7f3d29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:29.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.220+0000 7f3d29ffb640 10 monclient: dump: 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:29.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.220+0000 7f3d2b7fe640 4 set_mon_vals no callback set 2024-03-25T19:44:29.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.221+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f3d2c003ba0 con 0x7f3d380b1730 2024-03-25T19:44:29.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.221+0000 7f3d3ca35640 10 monclient: get_auth_request con 0x7f3d1c06dd10 auth_method 0 2024-03-25T19:44:29.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.222+0000 7f3d29ffb640 10 client.14402.objecter ms_handle_connect 0x7f3d1c06dd10 2024-03-25T19:44:29.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.227+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f3d2c054780 con 0x7f3d380b1730 2024-03-25T19:44:29.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.227+0000 7f3d29ffb640 10 client.14402.objecter ms_dispatch 0x7f3d38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:29.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.227+0000 7f3d29ffb640 3 client.14402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:29.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.227+0000 7f3d29ffb640 3 client.14402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:29.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.227+0000 7f3d29ffb640 20 client.14402.objecter dump_active .. 0 homeless 2024-03-25T19:44:29.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.227+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d2c050290 con 0x7f3d380b1730 2024-03-25T19:44:29.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.227+0000 7f3d29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:29.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.227+0000 7f3d29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:29.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.471+0000 7f3d3f49a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:29.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.471+0000 7f3d3f49a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:29.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.471+0000 7f3d3f49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:44:29.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.471+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d380627b0 con 0x7f3d380b1730 2024-03-25T19:44:29.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.504+0000 7f3d29ffb640 1 -- 172.21.15.71:0/1322144240 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f3d2c0521c0 con 0x7f3d380b1730 2024-03-25T19:44:29.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.504+0000 7f3d29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:29.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.504+0000 7f3d29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:29.505 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:44:29.505 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:29.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 10 librados: watch_flush enter 2024-03-25T19:44:29.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 10 librados: watch_flush exit 2024-03-25T19:44:29.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 20 client.14402.objecter shutdown clearing up homeless session... 2024-03-25T19:44:29.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 10 client.14402.objecter successfully canceled tick 2024-03-25T19:44:29.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 >> v1:172.21.15.71:6800/3911735624 conn(0x7f3d1c06dd10 legacy=0x7f3d1c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 10 monclient: shutdown 2024-03-25T19:44:29.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:29.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 >> v1:172.21.15.71:6790/0 conn(0x7f3d380b1730 legacy=0x7f3d38131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 shutdown_connections 2024-03-25T19:44:29.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 >> 172.21.15.71:0/1322144240 conn(0x7f3d380617a0 msgr2=0x7f3d380b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:29.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 shutdown_connections 2024-03-25T19:44:29.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 1 -- 172.21.15.71:0/1322144240 wait complete. 2024-03-25T19:44:29.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:29.507+0000 7f3d3f49a640 1 librados: shutdown 2024-03-25T19:44:29.519 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:36.326 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:36.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.442+0000 7f485aad4640 10 monclient: get_monmap_and_config 2024-03-25T19:44:36.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.442+0000 7f485aad4640 10 monclient: build_initial_monmap 2024-03-25T19:44:36.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.442+0000 7f485aad4640 10 monclient: monmap: 2024-03-25T19:44:36.442 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:44:36.443635+0000 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:44:36.443635+0000 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:44:36.443 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:36.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.443+0000 7f485aad4640 1 Processor -- start 2024-03-25T19:44:36.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.443+0000 7f485aad4640 1 -- start start 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.443+0000 7f485aad4640 10 monclient: init 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 10 monclient(hunting): picked mon.noname-c con 0x7f48540a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 10 monclient(hunting): picked mon.noname-a con 0x7f485413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 10 monclient(hunting): picked mon.noname-b con 0x7f48541430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4854142520 con 0x7f485413f710 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48540b28a0 con 0x7f48540a98f0 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48540b2a30 con 0x7f48541430c0 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:36.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.444+0000 7f485aad4640 10 monclient(hunting): authenticate will time out at 5082.756526s 2024-03-25T19:44:36.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.448+0000 7f4859071640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f48541430c0 0x7f485414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39872/0 (socket says 172.21.15.71:39872) 2024-03-25T19:44:36.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.448+0000 7f4859071640 1 -- 172.21.15.71:0/3489402008 learned_addr learned my addr 172.21.15.71:0/3489402008 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:36.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.459+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937626269 0 0) 0x7f48540b28a0 con 0x7f48540a98f0 2024-03-25T19:44:36.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.459+0000 7f484b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.459+0000 7f484b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.459+0000 7f484b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.459+0000 7f484b7fe640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:44:36.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.459+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f483c003520 con 0x7f48540a98f0 2024-03-25T19:44:36.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.465+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817121974 0 0) 0x7f48540b2a30 con 0x7f48541430c0 2024-03-25T19:44:36.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.465+0000 7f484b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.465+0000 7f484b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.465+0000 7f484b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.465+0000 7f484b7fe640 10 monclient(hunting): my global_id is 14405 2024-03-25T19:44:36.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.465+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48540b28a0 con 0x7f48541430c0 2024-03-25T19:44:36.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.475+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364587068 0 0) 0x7f4854142520 con 0x7f485413f710 2024-03-25T19:44:36.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.475+0000 7f484b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.475+0000 7f484b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.475+0000 7f484b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.475+0000 7f484b7fe640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:44:36.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.475+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48540b2a30 con 0x7f485413f710 2024-03-25T19:44:36.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4198752923 0 0) 0x7f48540b2a30 con 0x7f485413f710 2024-03-25T19:44:36.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 >> v1:172.21.15.71:6790/0 conn(0x7f48541430c0 legacy=0x7f485414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 >> v1:172.21.15.193:6789/0 conn(0x7f48540a98f0 legacy=0x7f48540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48540b44c0 con 0x7f485413f710 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f48440015b0 con 0x7f485413f710 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.489+0000 7f484b7fe640 10 monclient: dump: 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.490+0000 7f485aad4640 5 monclient: authenticate success, global_id 4398 2024-03-25T19:44:36.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.490+0000 7f485aad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.526+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4844001710 con 0x7f485413f710 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.526+0000 7f484b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.526+0000 7f484b7fe640 1 -- 172.21.15.71:0/3489402008 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4844001b20 con 0x7f485413f710 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.526+0000 7f484b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.526+0000 7f484b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.526+0000 7f484b7fe640 10 monclient: dump: 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:36.526 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:36.527 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:36.527 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:36.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.526+0000 7f485aad4640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:36.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.526+0000 7f485aad4640 4 set_mon_vals no callback set 2024-03-25T19:44:36.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.527+0000 7f485aad4640 10 monclient: shutdown 2024-03-25T19:44:36.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.527+0000 7f485aad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:36.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.527+0000 7f485aad4640 1 -- 172.21.15.71:0/3489402008 >> v1:172.21.15.71:6789/0 conn(0x7f485413f710 legacy=0x7f48541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.527+0000 7f485aad4640 1 -- 172.21.15.71:0/3489402008 shutdown_connections 2024-03-25T19:44:36.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.527+0000 7f485aad4640 1 -- 172.21.15.71:0/3489402008 >> 172.21.15.71:0/3489402008 conn(0x7f48540b7ea0 msgr2=0x7f48540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:36.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.527+0000 7f485aad4640 1 -- 172.21.15.71:0/3489402008 shutdown_connections 2024-03-25T19:44:36.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.527+0000 7f485aad4640 1 -- 172.21.15.71:0/3489402008 wait complete. 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 10 monclient: build_initial_monmap 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 10 monclient: monmap: 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 1 librados: starting msgr at 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 1 librados: starting objecter 2024-03-25T19:44:36.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 1 Processor -- start 2024-03-25T19:44:36.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 1 -- start start 2024-03-25T19:44:36.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 1 librados: setting wanted keys 2024-03-25T19:44:36.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 1 librados: calling monclient init 2024-03-25T19:44:36.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.528+0000 7f485aad4640 10 monclient: init 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.529+0000 7f485aad4640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.529+0000 7f485aad4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.529+0000 7f485aad4640 10 monclient(hunting): picked mon.noname-b con 0x7f48540a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.529+0000 7f485aad4640 10 monclient(hunting): picked mon.noname-a con 0x7f485413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.529+0000 7f485aad4640 10 monclient(hunting): picked mon.noname-c con 0x7f48541430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.530+0000 7f485aad4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4854138ff0 con 0x7f485413f710 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.530+0000 7f485aad4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48541374f0 con 0x7f48540a98f0 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.530+0000 7f485aad4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48541db2f0 con 0x7f48541430c0 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.530+0000 7f485aad4640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:36.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.530+0000 7f485aad4640 10 monclient(hunting): authenticate will time out at 5082.841954s 2024-03-25T19:44:36.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.536+0000 7f484bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f485413f710 0x7f4854132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:45828/0 (socket says 172.21.15.71:45828) 2024-03-25T19:44:36.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.536+0000 7f484bfff640 1 -- 172.21.15.71:0/4175484325 learned_addr learned my addr 172.21.15.71:0/4175484325 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:36.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.537+0000 7f48497fa640 10 client.?.objecter ms_handle_connect 0x7f485413f710 2024-03-25T19:44:36.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.537+0000 7f48497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:36.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.545+0000 7f48497fa640 10 client.?.objecter ms_handle_connect 0x7f48541430c0 2024-03-25T19:44:36.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.545+0000 7f48497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:36.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.550+0000 7f48497fa640 10 client.?.objecter ms_handle_connect 0x7f48540a98f0 2024-03-25T19:44:36.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.550+0000 7f48497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:36.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.550+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588351506 0 0) 0x7f4854138ff0 con 0x7f485413f710 2024-03-25T19:44:36.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.550+0000 7f48497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.550+0000 7f48497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.550+0000 7f48497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.550+0000 7f48497fa640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:44:36.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.550+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840004eb0 con 0x7f485413f710 2024-03-25T19:44:36.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.554+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105135975 0 0) 0x7f48541374f0 con 0x7f48540a98f0 2024-03-25T19:44:36.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.554+0000 7f48497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.554+0000 7f48497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.554+0000 7f48497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.554+0000 7f48497fa640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:44:36.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.554+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4854138ff0 con 0x7f48540a98f0 2024-03-25T19:44:36.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.555+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402618256 0 0) 0x7f48541db2f0 con 0x7f48541430c0 2024-03-25T19:44:36.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.555+0000 7f48497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.555+0000 7f48497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.555+0000 7f48497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.555+0000 7f48497fa640 10 monclient(hunting): my global_id is 14408 2024-03-25T19:44:36.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.555+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48541374f0 con 0x7f48541430c0 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.567+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3983803010 0 0) 0x7f4854138ff0 con 0x7f48540a98f0 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.567+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48541db2f0 con 0x7f48540a98f0 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.567+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f48500012f0 con 0x7f48540a98f0 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.567+0000 7f48497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.567+0000 7f48497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.567+0000 7f48497fa640 10 monclient(hunting): dump: 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:36.567 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:36.568 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:36.568 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:36.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.567+0000 7f48497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.569+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3508246044 0 0) 0x7f4840004eb0 con 0x7f485413f710 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.569+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4854138ff0 con 0x7f485413f710 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.569+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4844001ae0 con 0x7f485413f710 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.569+0000 7f48497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.569+0000 7f48497fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.569+0000 7f48497fa640 10 monclient(hunting): dump: 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.569 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.570 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:36.570 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:36.570 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:36.570 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:36.570 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:36.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2918633556 0 0) 0x7f48541db2f0 con 0x7f48540a98f0 2024-03-25T19:44:36.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 >> v1:172.21.15.71:6790/0 conn(0x7f48541430c0 legacy=0x7f48541e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 >> v1:172.21.15.71:6789/0 conn(0x7f485413f710 legacy=0x7f4854132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:36.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 1 monclient: found mon.b 2024-03-25T19:44:36.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:36.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:36.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4854137a10 con 0x7f48540a98f0 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f48497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 10 monclient: _renew_subs 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4854137c60 con 0x7f48540a98f0 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 10 monclient: _renew_subs 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4854138120 con 0x7f48540a98f0 2024-03-25T19:44:36.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.579+0000 7f485aad4640 1 librados: init done 2024-03-25T19:44:36.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.580+0000 7f485aad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:36.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.580+0000 7f485aad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:36.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.580+0000 7f485aad4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:36.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.580+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48540b3ef0 con 0x7f48540a98f0 2024-03-25T19:44:36.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.859+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48500035f0 con 0x7f48540a98f0 2024-03-25T19:44:36.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.859+0000 7f48497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.859+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4850003900 con 0x7f48540a98f0 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.859+0000 7f48497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.859+0000 7f48497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.859+0000 7f48497fa640 10 monclient: dump: 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:36.860 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:36.861 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:36.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.859+0000 7f484affd640 4 set_mon_vals no callback set 2024-03-25T19:44:36.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.860+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f48500198d0 con 0x7f48540a98f0 2024-03-25T19:44:36.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.860+0000 7f484bfff640 10 monclient: get_auth_request con 0x7f484006dec0 auth_method 0 2024-03-25T19:44:36.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.861+0000 7f48497fa640 10 client.4402.objecter ms_handle_connect 0x7f484006dec0 2024-03-25T19:44:36.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.865+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f4850053540 con 0x7f48540a98f0 2024-03-25T19:44:36.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.865+0000 7f48497fa640 10 client.4402.objecter ms_dispatch 0x7f4854000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:36.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.865+0000 7f48497fa640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:36.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.865+0000 7f48497fa640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:36.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.866+0000 7f48497fa640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T19:44:36.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.867+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f485004ee80 con 0x7f48540a98f0 2024-03-25T19:44:36.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.867+0000 7f48497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:36.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:36.867+0000 7f48497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:37.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.091+0000 7f485aad4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:37.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.091+0000 7f485aad4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:37.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.091+0000 7f485aad4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:37.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.091+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4854137680 con 0x7f48540a98f0 2024-03-25T19:44:37.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.119+0000 7f48497fa640 1 -- 172.21.15.71:0/4175484325 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f4850056ba0 con 0x7f48540a98f0 2024-03-25T19:44:37.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.119+0000 7f48497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:37.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.119+0000 7f48497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:37.119 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:44:37.119 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:37.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.121+0000 7f485aad4640 10 librados: watch_flush enter 2024-03-25T19:44:37.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.121+0000 7f485aad4640 10 librados: watch_flush exit 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.121+0000 7f485aad4640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.121+0000 7f485aad4640 10 client.4402.objecter successfully canceled tick 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 >> v1:172.21.15.71:6800/3911735624 conn(0x7f484006dec0 legacy=0x7f4840070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 10 monclient: shutdown 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 >> v1:172.21.15.193:6789/0 conn(0x7f48540a98f0 legacy=0x7f48541dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 shutdown_connections 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 >> 172.21.15.71:0/4175484325 conn(0x7f48540b7ea0 msgr2=0x7f4854074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 shutdown_connections 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 1 -- 172.21.15.71:0/4175484325 wait complete. 2024-03-25T19:44:37.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:37.122+0000 7f485aad4640 1 librados: shutdown 2024-03-25T19:44:37.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:43.940 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.037+0000 7f3db213b640 10 monclient: get_monmap_and_config 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.037+0000 7f3db213b640 10 monclient: build_initial_monmap 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.037+0000 7f3db213b640 10 monclient: monmap: 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:44:44.039246+0000 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:44:44.039246+0000 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:44.038 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:44.039 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:44.039 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:44:44.039 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:44:44.039 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:44.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.038+0000 7f3db213b640 1 Processor -- start 2024-03-25T19:44:44.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.038+0000 7f3db213b640 1 -- start start 2024-03-25T19:44:44.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.038+0000 7f3db213b640 10 monclient: init 2024-03-25T19:44:44.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.038+0000 7f3db213b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:44.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.038+0000 7f3db213b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:44.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.038+0000 7f3db213b640 10 monclient(hunting): picked mon.noname-b con 0x7f3dac0b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:44.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.038+0000 7f3db213b640 10 monclient(hunting): picked mon.noname-a con 0x7f3dac0b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:44.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.039+0000 7f3db213b640 10 monclient(hunting): picked mon.noname-c con 0x7f3dac14be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:44.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.039+0000 7f3db213b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac0af940 con 0x7f3dac0b1730 2024-03-25T19:44:44.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.039+0000 7f3db213b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac0b0a70 con 0x7f3dac14be90 2024-03-25T19:44:44.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.039+0000 7f3db213b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac0b0d50 con 0x7f3dac0b4d60 2024-03-25T19:44:44.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.039+0000 7f3db213b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:44.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.039+0000 7f3db213b640 10 monclient(hunting): authenticate will time out at 5090.352001s 2024-03-25T19:44:44.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.054+0000 7f3daaffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3dac0b1730 0x7f3dac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50484/0 (socket says 172.21.15.71:50484) 2024-03-25T19:44:44.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.054+0000 7f3daaffd640 1 -- 172.21.15.71:0/1348799138 learned_addr learned my addr 172.21.15.71:0/1348799138 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:44.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.066+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321683914 0 0) 0x7f3dac0af940 con 0x7f3dac0b1730 2024-03-25T19:44:44.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.066+0000 7f3daa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:44.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.066+0000 7f3daa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:44.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.066+0000 7f3daa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:44.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.066+0000 7f3daa7fc640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:44:44.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.066+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d94003520 con 0x7f3dac0b1730 2024-03-25T19:44:44.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.072+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902479928 0 0) 0x7f3dac0b0a70 con 0x7f3dac14be90 2024-03-25T19:44:44.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.072+0000 7f3daa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:44.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.072+0000 7f3daa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:44.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.072+0000 7f3daa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:44.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.072+0000 7f3daa7fc640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:44:44.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.072+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac0af940 con 0x7f3dac14be90 2024-03-25T19:44:44.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.073+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28502183 0 0) 0x7f3dac0b0d50 con 0x7f3dac0b4d60 2024-03-25T19:44:44.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.073+0000 7f3daa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:44.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.073+0000 7f3daa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:44.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.073+0000 7f3daa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:44.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.073+0000 7f3daa7fc640 10 monclient(hunting): my global_id is 14411 2024-03-25T19:44:44.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.073+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac0b0a70 con 0x7f3dac0b4d60 2024-03-25T19:44:44.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1550186583 0 0) 0x7f3dac0b0a70 con 0x7f3dac0b4d60 2024-03-25T19:44:44.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 >> v1:172.21.15.193:6789/0 conn(0x7f3dac14be90 legacy=0x7f3dac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 >> v1:172.21.15.71:6789/0 conn(0x7f3dac0b1730 legacy=0x7f3dac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:44.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 1 monclient: found mon.noname-b 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac0b1030 con 0x7f3dac0b4d60 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3d9c0015b0 con 0x7f3dac0b4d60 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3daa7fc640 10 monclient: dump: 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:44.086 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3db213b640 5 monclient: authenticate success, global_id 14411 2024-03-25T19:44:44.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.084+0000 7f3db213b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.121+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d9c001710 con 0x7f3dac0b4d60 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.121+0000 7f3daa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.121+0000 7f3daa7fc640 1 -- 172.21.15.71:0/1348799138 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3d9c001b60 con 0x7f3dac0b4d60 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.121+0000 7f3daa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.121+0000 7f3daa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.121+0000 7f3daa7fc640 10 monclient: dump: 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.122 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:44.123 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:44.123 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:44.123 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:44.123 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:44.123 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:44.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.121+0000 7f3db213b640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:44.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.121+0000 7f3db213b640 4 set_mon_vals no callback set 2024-03-25T19:44:44.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.122+0000 7f3db213b640 10 monclient: shutdown 2024-03-25T19:44:44.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.122+0000 7f3db213b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:44.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.122+0000 7f3db213b640 1 -- 172.21.15.71:0/1348799138 >> v1:172.21.15.71:6790/0 conn(0x7f3dac0b4d60 legacy=0x7f3dac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.122+0000 7f3db213b640 1 -- 172.21.15.71:0/1348799138 shutdown_connections 2024-03-25T19:44:44.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.122+0000 7f3db213b640 1 -- 172.21.15.71:0/1348799138 >> 172.21.15.71:0/1348799138 conn(0x7f3dac0617a0 msgr2=0x7f3dac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.122+0000 7f3db213b640 1 -- 172.21.15.71:0/1348799138 shutdown_connections 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.122+0000 7f3db213b640 1 -- 172.21.15.71:0/1348799138 wait complete. 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.123+0000 7f3db213b640 10 monclient: build_initial_monmap 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.123+0000 7f3db213b640 10 monclient: monmap: 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:44:44.125 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:44.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.124+0000 7f3db213b640 1 librados: starting msgr at 2024-03-25T19:44:44.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.124+0000 7f3db213b640 1 librados: starting objecter 2024-03-25T19:44:44.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.124+0000 7f3db213b640 1 Processor -- start 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.124+0000 7f3db213b640 1 -- start start 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.124+0000 7f3db213b640 1 librados: setting wanted keys 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.124+0000 7f3db213b640 1 librados: calling monclient init 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.124+0000 7f3db213b640 10 monclient: init 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 10 monclient(hunting): picked mon.noname-b con 0x7f3dac0b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 10 monclient(hunting): picked mon.noname-c con 0x7f3dac0b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 10 monclient(hunting): picked mon.noname-a con 0x7f3dac14be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac1409c0 con 0x7f3dac14be90 2024-03-25T19:44:44.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac141d10 con 0x7f3dac0b1730 2024-03-25T19:44:44.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac140e50 con 0x7f3dac0b4d60 2024-03-25T19:44:44.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:44.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.125+0000 7f3db213b640 10 monclient(hunting): authenticate will time out at 5090.438216s 2024-03-25T19:44:44.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.138+0000 7f3dabfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3dac14be90 0x7f3dac1e2ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50530/0 (socket says 172.21.15.71:50530) 2024-03-25T19:44:44.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.138+0000 7f3dabfff640 1 -- 172.21.15.71:0/1479229393 learned_addr learned my addr 172.21.15.71:0/1479229393 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:44.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.139+0000 7f3d93fff640 10 client.?.objecter ms_handle_connect 0x7f3dac0b4d60 2024-03-25T19:44:44.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.139+0000 7f3d93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:44.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.142+0000 7f3d93fff640 10 client.?.objecter ms_handle_connect 0x7f3dac14be90 2024-03-25T19:44:44.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.142+0000 7f3d93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:44.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.142+0000 7f3d93fff640 10 client.?.objecter ms_handle_connect 0x7f3dac0b1730 2024-03-25T19:44:44.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.142+0000 7f3d93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:44.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.159+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841924565 0 0) 0x7f3dac141d10 con 0x7f3dac0b1730 2024-03-25T19:44:44.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.159+0000 7f3d93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:44.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.159+0000 7f3d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:44.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.159+0000 7f3d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:44.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.159+0000 7f3d93fff640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:44:44.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.159+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d98004eb0 con 0x7f3dac0b1730 2024-03-25T19:44:44.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.167+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071230735 0 0) 0x7f3dac1409c0 con 0x7f3dac14be90 2024-03-25T19:44:44.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.167+0000 7f3d93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:44.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.167+0000 7f3d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:44.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.167+0000 7f3d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:44.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.167+0000 7f3d93fff640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:44:44.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.167+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac141d10 con 0x7f3dac14be90 2024-03-25T19:44:44.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.169+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914993214 0 0) 0x7f3dac140e50 con 0x7f3dac0b4d60 2024-03-25T19:44:44.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.169+0000 7f3d93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:44.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.169+0000 7f3d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:44.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.169+0000 7f3d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:44.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.169+0000 7f3d93fff640 10 monclient(hunting): my global_id is 14414 2024-03-25T19:44:44.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.169+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac1409c0 con 0x7f3dac0b4d60 2024-03-25T19:44:44.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.182+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 676108665 0 0) 0x7f3dac141d10 con 0x7f3dac14be90 2024-03-25T19:44:44.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.183+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dac140e50 con 0x7f3dac14be90 2024-03-25T19:44:44.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.183+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3da4001c10 con 0x7f3dac14be90 2024-03-25T19:44:44.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.183+0000 7f3d93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:44.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.183+0000 7f3d93fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.183+0000 7f3d93fff640 10 monclient(hunting): dump: 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:44.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.183+0000 7f3d93fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:44.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.185+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717968472 0 0) 0x7f3d98004eb0 con 0x7f3dac0b1730 2024-03-25T19:44:44.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.185+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dac141d10 con 0x7f3dac0b1730 2024-03-25T19:44:44.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.185+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3d9c001a20 con 0x7f3dac0b1730 2024-03-25T19:44:44.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.185+0000 7f3d93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:44.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.185+0000 7f3d93fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.185+0000 7f3d93fff640 10 monclient(hunting): dump: 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:44.187 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:44.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.190+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2728695343 0 0) 0x7f3dac1409c0 con 0x7f3dac0b4d60 2024-03-25T19:44:44.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.190+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d98004eb0 con 0x7f3dac0b4d60 2024-03-25T19:44:44.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.190+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3da00018d0 con 0x7f3dac0b4d60 2024-03-25T19:44:44.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.190+0000 7f3d93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:44.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.190+0000 7f3d93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.190+0000 7f3d93fff640 10 monclient(hunting): dump: 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:44.192 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:44.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3161776182 0 0) 0x7f3dac140e50 con 0x7f3dac14be90 2024-03-25T19:44:44.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 >> v1:172.21.15.71:6790/0 conn(0x7f3dac0b4d60 legacy=0x7f3dac1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 >> v1:172.21.15.193:6789/0 conn(0x7f3dac0b1730 legacy=0x7f3dac131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 1 monclient: found mon.a 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac132520 con 0x7f3dac14be90 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.197+0000 7f3d93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 5 monclient: authenticate success, global_id 4407 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 10 monclient: _renew_subs 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dac132770 con 0x7f3dac14be90 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 10 monclient: _renew_subs 2024-03-25T19:44:44.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:44.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dac138120 con 0x7f3dac14be90 2024-03-25T19:44:44.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.198+0000 7f3db213b640 1 librados: init done 2024-03-25T19:44:44.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.199+0000 7f3db213b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:44.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.199+0000 7f3db213b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:44.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.199+0000 7f3db213b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:44.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.199+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dac0627b0 con 0x7f3dac14be90 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.508+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3da4003990 con 0x7f3dac14be90 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.508+0000 7f3d93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.508+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3da4002100 con 0x7f3dac14be90 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.508+0000 7f3d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.508+0000 7f3d93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.508+0000 7f3d93fff640 10 monclient: dump: 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.509 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:44.510 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:44.510 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:44.510 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:44.510 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:44.510 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:44.510 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:44.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.508+0000 7f3da9ffb640 4 set_mon_vals no callback set 2024-03-25T19:44:44.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.509+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f3da4003de0 con 0x7f3dac14be90 2024-03-25T19:44:44.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.509+0000 7f3dab7fe640 10 monclient: get_auth_request con 0x7f3d9806e0b0 auth_method 0 2024-03-25T19:44:44.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.509+0000 7f3d93fff640 10 client.4407.objecter ms_handle_connect 0x7f3d9806e0b0 2024-03-25T19:44:44.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.516+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f3da40047e0 con 0x7f3dac14be90 2024-03-25T19:44:44.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.516+0000 7f3d93fff640 10 client.4407.objecter ms_dispatch 0x7f3dac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:44.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.516+0000 7f3d93fff640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:44.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.516+0000 7f3d93fff640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:44.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.516+0000 7f3d93fff640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-25T19:44:44.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.516+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3da4050430 con 0x7f3dac14be90 2024-03-25T19:44:44.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.516+0000 7f3d93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:44.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.516+0000 7f3d93fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:44.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.769+0000 7f3db213b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:44.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.769+0000 7f3db213b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:44.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.769+0000 7f3db213b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:44.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.769+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dac069f40 con 0x7f3dac14be90 2024-03-25T19:44:44.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.800+0000 7f3d93fff640 1 -- 172.21.15.71:0/1479229393 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f3da4052360 con 0x7f3dac14be90 2024-03-25T19:44:44.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.800+0000 7f3d93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:44.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.800+0000 7f3d93fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:44.801 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:44:44.801 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:44.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 10 librados: watch_flush enter 2024-03-25T19:44:44.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 10 librados: watch_flush exit 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 10 client.4407.objecter successfully canceled tick 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 >> v1:172.21.15.71:6800/3911735624 conn(0x7f3d9806e0b0 legacy=0x7f3d98070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 10 monclient: shutdown 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 >> v1:172.21.15.71:6789/0 conn(0x7f3dac14be90 legacy=0x7f3dac1e2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 shutdown_connections 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 >> 172.21.15.71:0/1479229393 conn(0x7f3dac0617a0 msgr2=0x7f3dac0b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 shutdown_connections 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 1 -- 172.21.15.71:0/1479229393 wait complete. 2024-03-25T19:44:44.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:44.802+0000 7f3db213b640 1 librados: shutdown 2024-03-25T19:44:44.814 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:51.620 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:51.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.740+0000 7f7b864b3640 10 monclient: get_monmap_and_config 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.740+0000 7f7b864b3640 10 monclient: build_initial_monmap 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.740+0000 7f7b864b3640 10 monclient: monmap: 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:44:51.740963+0000 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:44:51.740963+0000 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:44:51.740 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:51.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.740+0000 7f7b864b3640 1 Processor -- start 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.740+0000 7f7b864b3640 1 -- start start 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.740+0000 7f7b864b3640 10 monclient: init 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 10 monclient(hunting): picked mon.noname-b con 0x7f7b800a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 10 monclient(hunting): picked mon.noname-a con 0x7f7b8013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 10 monclient(hunting): picked mon.noname-c con 0x7f7b801430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b80142520 con 0x7f7b8013f710 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b800b28a0 con 0x7f7b801430c0 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b800b2a30 con 0x7f7b800a98f0 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:51.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.741+0000 7f7b864b3640 10 monclient(hunting): authenticate will time out at 5098.053366s 2024-03-25T19:44:51.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.749+0000 7f7b84a50640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7b801430c0 0x7f7b8014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:45204/0 (socket says 172.21.15.71:45204) 2024-03-25T19:44:51.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.749+0000 7f7b84a50640 1 -- 172.21.15.71:0/3632836403 learned_addr learned my addr 172.21.15.71:0/3632836403 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:51.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.768+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610259224 0 0) 0x7f7b800b28a0 con 0x7f7b801430c0 2024-03-25T19:44:51.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.768+0000 7f7b7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.768+0000 7f7b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.768+0000 7f7b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.768+0000 7f7b7effd640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:44:51.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.768+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b68003520 con 0x7f7b801430c0 2024-03-25T19:44:51.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.772+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136438932 0 0) 0x7f7b80142520 con 0x7f7b8013f710 2024-03-25T19:44:51.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.772+0000 7f7b7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.772+0000 7f7b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.772+0000 7f7b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.772+0000 7f7b7effd640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:44:51.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.772+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b800b28a0 con 0x7f7b8013f710 2024-03-25T19:44:51.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172980433 0 0) 0x7f7b68003520 con 0x7f7b801430c0 2024-03-25T19:44:51.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 >> v1:172.21.15.71:6790/0 conn(0x7f7b800a98f0 legacy=0x7f7b800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 >> v1:172.21.15.71:6789/0 conn(0x7f7b8013f710 legacy=0x7f7b801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 1 monclient: found mon.noname-c 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b800b44c0 con 0x7f7b801430c0 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 10 monclient: _finish_auth 0 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b78001a10 con 0x7f7b801430c0 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b7effd640 10 monclient: dump: 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:51.777 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b864b3640 5 monclient: authenticate success, global_id 4411 2024-03-25T19:44:51.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.776+0000 7f7b864b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:51.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b78001b70 con 0x7f7b801430c0 2024-03-25T19:44:51.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:51.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b7effd640 1 -- 172.21.15.71:0/3632836403 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b78001f80 con 0x7f7b801430c0 2024-03-25T19:44:51.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b864b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b864b3640 4 set_mon_vals no callback set 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b7effd640 10 monclient: dump: 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:51.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b864b3640 10 monclient: shutdown 2024-03-25T19:44:51.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b864b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:51.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b864b3640 1 -- 172.21.15.71:0/3632836403 >> v1:172.21.15.193:6789/0 conn(0x7f7b801430c0 legacy=0x7f7b8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b864b3640 1 -- 172.21.15.71:0/3632836403 shutdown_connections 2024-03-25T19:44:51.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.779+0000 7f7b864b3640 1 -- 172.21.15.71:0/3632836403 >> 172.21.15.71:0/3632836403 conn(0x7f7b800b7ea0 msgr2=0x7f7b800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:51.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.780+0000 7f7b864b3640 1 -- 172.21.15.71:0/3632836403 shutdown_connections 2024-03-25T19:44:51.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.780+0000 7f7b864b3640 1 -- 172.21.15.71:0/3632836403 wait complete. 2024-03-25T19:44:51.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.780+0000 7f7b864b3640 10 monclient: build_initial_monmap 2024-03-25T19:44:51.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.780+0000 7f7b864b3640 10 monclient: monmap: 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.780+0000 7f7b864b3640 1 librados: starting msgr at 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.780+0000 7f7b864b3640 1 librados: starting objecter 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 1 Processor -- start 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 1 -- start start 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 1 librados: setting wanted keys 2024-03-25T19:44:51.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 1 librados: calling monclient init 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 10 monclient: init 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 10 monclient(hunting): picked mon.noname-b con 0x7f7b800a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 10 monclient(hunting): picked mon.noname-c con 0x7f7b8013f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 10 monclient(hunting): picked mon.noname-a con 0x7f7b801430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b801db170 con 0x7f7b801430c0 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b801dac50 con 0x7f7b800a98f0 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b801dade0 con 0x7f7b8013f710 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:51.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.781+0000 7f7b864b3640 10 monclient(hunting): authenticate will time out at 5098.093737s 2024-03-25T19:44:51.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.789+0000 7f7b7f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7b8013f710 0x7f7b801d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:60460/0 (socket says 172.21.15.71:60460) 2024-03-25T19:44:51.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.789+0000 7f7b7f7fe640 1 -- 172.21.15.71:0/7441067 learned_addr learned my addr 172.21.15.71:0/7441067 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:51.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.792+0000 7f7b7cff9640 10 client.?.objecter ms_handle_connect 0x7f7b800a98f0 2024-03-25T19:44:51.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.793+0000 7f7b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:51.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.796+0000 7f7b7cff9640 10 client.?.objecter ms_handle_connect 0x7f7b8013f710 2024-03-25T19:44:51.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.796+0000 7f7b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:51.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.809+0000 7f7b7cff9640 10 client.?.objecter ms_handle_connect 0x7f7b801430c0 2024-03-25T19:44:51.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.809+0000 7f7b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:51.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.810+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190807306 0 0) 0x7f7b801dade0 con 0x7f7b8013f710 2024-03-25T19:44:51.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.810+0000 7f7b7cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.810+0000 7f7b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.810+0000 7f7b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.810+0000 7f7b7cff9640 10 monclient(hunting): my global_id is 14420 2024-03-25T19:44:51.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.810+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c004eb0 con 0x7f7b8013f710 2024-03-25T19:44:51.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.812+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846883192 0 0) 0x7f7b801dac50 con 0x7f7b800a98f0 2024-03-25T19:44:51.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.812+0000 7f7b7cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.812+0000 7f7b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.812+0000 7f7b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.812+0000 7f7b7cff9640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:44:51.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.812+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b801dade0 con 0x7f7b800a98f0 2024-03-25T19:44:51.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.824+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1531524676 0 0) 0x7f7b6c004eb0 con 0x7f7b8013f710 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.824+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b801dac50 con 0x7f7b8013f710 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.824+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b74001a10 con 0x7f7b8013f710 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.824+0000 7f7b7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.824+0000 7f7b7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.824+0000 7f7b7cff9640 10 monclient(hunting): dump: 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:51.824 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:51.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.824+0000 7f7b7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:51.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.825+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189566420 0 0) 0x7f7b801dade0 con 0x7f7b800a98f0 2024-03-25T19:44:51.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.825+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6c004eb0 con 0x7f7b800a98f0 2024-03-25T19:44:51.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.825+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b700018a0 con 0x7f7b800a98f0 2024-03-25T19:44:51.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.825+0000 7f7b7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.825+0000 7f7b7cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.825+0000 7f7b7cff9640 10 monclient(hunting): dump: 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:51.826 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:51.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.830+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837260009 0 0) 0x7f7b801db170 con 0x7f7b801430c0 2024-03-25T19:44:51.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.830+0000 7f7b7cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.830+0000 7f7b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.830+0000 7f7b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.830+0000 7f7b7cff9640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:44:51.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.830+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b801dade0 con 0x7f7b801430c0 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3768470362 0 0) 0x7f7b6c004eb0 con 0x7f7b800a98f0 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 >> v1:172.21.15.71:6790/0 conn(0x7f7b8013f710 legacy=0x7f7b801d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 >> v1:172.21.15.71:6789/0 conn(0x7f7b801430c0 legacy=0x7f7b80132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 1 monclient: found mon.b 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8013c170 con 0x7f7b800a98f0 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 5 monclient: authenticate success, global_id 4414 2024-03-25T19:44:51.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 10 monclient: _renew_subs 2024-03-25T19:44:51.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:51.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8013c420 con 0x7f7b800a98f0 2024-03-25T19:44:51.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:51.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 10 monclient: _renew_subs 2024-03-25T19:44:51.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:51.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b80133860 con 0x7f7b800a98f0 2024-03-25T19:44:51.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.833+0000 7f7b864b3640 1 librados: init done 2024-03-25T19:44:51.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.834+0000 7f7b864b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:51.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.834+0000 7f7b864b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:51.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.834+0000 7f7b864b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:51.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:51.834+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b801daf70 con 0x7f7b800a98f0 2024-03-25T19:44:52.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.106+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b700035f0 con 0x7f7b800a98f0 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.106+0000 7f7b7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.106+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b700039c0 con 0x7f7b800a98f0 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.106+0000 7f7b7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.106+0000 7f7b7cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.106+0000 7f7b7cff9640 10 monclient: dump: 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:52.106 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:52.107 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:52.107 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:52.107 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:52.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.106+0000 7f7b7e7fc640 4 set_mon_vals no callback set 2024-03-25T19:44:52.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.106+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f7b70003be0 con 0x7f7b800a98f0 2024-03-25T19:44:52.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.107+0000 7f7b7f7fe640 10 monclient: get_auth_request con 0x7f7b6c06df30 auth_method 0 2024-03-25T19:44:52.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.107+0000 7f7b7cff9640 10 client.4414.objecter ms_handle_connect 0x7f7b6c06df30 2024-03-25T19:44:52.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.111+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f7b70053590 con 0x7f7b800a98f0 2024-03-25T19:44:52.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.111+0000 7f7b7cff9640 10 client.4414.objecter ms_dispatch 0x7f7b80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:52.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.111+0000 7f7b7cff9640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:52.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.111+0000 7f7b7cff9640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:52.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.111+0000 7f7b7cff9640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T19:44:52.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.111+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b7004f0a0 con 0x7f7b800a98f0 2024-03-25T19:44:52.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.111+0000 7f7b7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:52.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.111+0000 7f7b7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:52.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.329+0000 7f7b864b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:52.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.329+0000 7f7b864b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:52.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.329+0000 7f7b864b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:44:52.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.329+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b800627b0 con 0x7f7b800a98f0 2024-03-25T19:44:52.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.354+0000 7f7b7cff9640 1 -- 172.21.15.71:0/7441067 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f7b70050fd0 con 0x7f7b800a98f0 2024-03-25T19:44:52.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.354+0000 7f7b7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:52.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.354+0000 7f7b7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:52.354 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:44:52.354 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 10 librados: watch_flush enter 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 10 librados: watch_flush exit 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 10 client.4414.objecter successfully canceled tick 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 >> v1:172.21.15.71:6800/3911735624 conn(0x7f7b6c06df30 legacy=0x7f7b6c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 10 monclient: shutdown 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 >> v1:172.21.15.193:6789/0 conn(0x7f7b800a98f0 legacy=0x7f7b801394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 shutdown_connections 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.356+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 >> 172.21.15.71:0/7441067 conn(0x7f7b800b7ea0 msgr2=0x7f7b8014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.357+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 shutdown_connections 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.357+0000 7f7b864b3640 1 -- 172.21.15.71:0/7441067 wait complete. 2024-03-25T19:44:52.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:52.357+0000 7f7b864b3640 1 librados: shutdown 2024-03-25T19:44:52.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:59.175 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.288+0000 7f6a3216c640 10 monclient: get_monmap_and_config 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.288+0000 7f6a3216c640 10 monclient: build_initial_monmap 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.288+0000 7f6a3216c640 10 monclient: monmap: 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:44:59.289275+0000 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:44:59.289275+0000 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:59.288 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:59.289 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:44:59.289 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:44:59.289 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:59.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.289+0000 7f6a3216c640 1 Processor -- start 2024-03-25T19:44:59.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.289+0000 7f6a3216c640 1 -- start start 2024-03-25T19:44:59.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.289+0000 7f6a3216c640 10 monclient: init 2024-03-25T19:44:59.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:59.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:59.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 10 monclient(hunting): picked mon.noname-a con 0x7f6a2c0a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:59.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 10 monclient(hunting): picked mon.noname-b con 0x7f6a2c13f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:59.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 10 monclient(hunting): picked mon.noname-c con 0x7f6a2c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:59.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c142520 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c0b28a0 con 0x7f6a2c1430c0 2024-03-25T19:44:59.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c0b2a30 con 0x7f6a2c13f710 2024-03-25T19:44:59.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:59.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.290+0000 7f6a3216c640 10 monclient(hunting): authenticate will time out at 5105.602335s 2024-03-25T19:44:59.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.302+0000 7f6a2b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f6a2c0a98f0 0x7f6a2c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52710/0 (socket says 172.21.15.71:52710) 2024-03-25T19:44:59.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.302+0000 7f6a2b7fe640 1 -- 172.21.15.71:0/3635465370 learned_addr learned my addr 172.21.15.71:0/3635465370 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:59.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.314+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424363871 0 0) 0x7f6a2c142520 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.314+0000 7f6a2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.314+0000 7f6a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.314+0000 7f6a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.314+0000 7f6a2a7fc640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:44:59.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.314+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a14003520 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.321+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932755492 0 0) 0x7f6a2c0b28a0 con 0x7f6a2c1430c0 2024-03-25T19:44:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.321+0000 7f6a2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.321+0000 7f6a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.321+0000 7f6a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.321+0000 7f6a2a7fc640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:44:59.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.321+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c142520 con 0x7f6a2c1430c0 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3749965692 0 0) 0x7f6a14003520 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 >> v1:172.21.15.71:6790/0 conn(0x7f6a2c13f710 legacy=0x7f6a2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 >> v1:172.21.15.193:6789/0 conn(0x7f6a2c1430c0 legacy=0x7f6a2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c0b44c0 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:59.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a1c001690 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a2a7fc640 10 monclient: dump: 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:59.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a3216c640 5 monclient: authenticate success, global_id 4416 2024-03-25T19:44:59.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.329+0000 7f6a3216c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:59.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.343+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a1c0017f0 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.343+0000 7f6a2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:59.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.343+0000 7f6a2a7fc640 1 -- 172.21.15.71:0/3635465370 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a1c001c00 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.343+0000 7f6a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.343+0000 7f6a2a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:59.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.343+0000 7f6a2a7fc640 10 monclient: dump: 2024-03-25T19:44:59.343 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:59.343 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.343+0000 7f6a3216c640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:59.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.343+0000 7f6a3216c640 4 set_mon_vals no callback set 2024-03-25T19:44:59.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.344+0000 7f6a3216c640 10 monclient: shutdown 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.344+0000 7f6a3216c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.344+0000 7f6a3216c640 1 -- 172.21.15.71:0/3635465370 >> v1:172.21.15.71:6789/0 conn(0x7f6a2c0a98f0 legacy=0x7f6a2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.344+0000 7f6a3216c640 1 -- 172.21.15.71:0/3635465370 shutdown_connections 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.344+0000 7f6a3216c640 1 -- 172.21.15.71:0/3635465370 >> 172.21.15.71:0/3635465370 conn(0x7f6a2c0b7ea0 msgr2=0x7f6a2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.344+0000 7f6a3216c640 1 -- 172.21.15.71:0/3635465370 shutdown_connections 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.344+0000 7f6a3216c640 1 -- 172.21.15.71:0/3635465370 wait complete. 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.345+0000 7f6a3216c640 10 monclient: build_initial_monmap 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.345+0000 7f6a3216c640 10 monclient: monmap: 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:44:59.346 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.345+0000 7f6a3216c640 1 librados: starting msgr at 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.345+0000 7f6a3216c640 1 librados: starting objecter 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.345+0000 7f6a3216c640 1 Processor -- start 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.346+0000 7f6a3216c640 1 -- start start 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.346+0000 7f6a3216c640 1 librados: setting wanted keys 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.346+0000 7f6a3216c640 1 librados: calling monclient init 2024-03-25T19:44:59.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.346+0000 7f6a3216c640 10 monclient: init 2024-03-25T19:44:59.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.346+0000 7f6a3216c640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:59.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.346+0000 7f6a3216c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:59.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.346+0000 7f6a3216c640 10 monclient(hunting): picked mon.noname-b con 0x7f6a2c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:44:59.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.347+0000 7f6a3216c640 10 monclient(hunting): picked mon.noname-a con 0x7f6a2c13f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:44:59.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.347+0000 7f6a3216c640 10 monclient(hunting): picked mon.noname-c con 0x7f6a2c1430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:44:59.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.347+0000 7f6a3216c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c1d41e0 con 0x7f6a2c13f710 2024-03-25T19:44:59.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.347+0000 7f6a3216c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c14e950 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.347+0000 7f6a3216c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c139840 con 0x7f6a2c1430c0 2024-03-25T19:44:59.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.347+0000 7f6a3216c640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:59.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.347+0000 7f6a3216c640 10 monclient(hunting): authenticate will time out at 5105.658954s 2024-03-25T19:44:59.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.359+0000 7f6a2affd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f6a2c13f710 0x7f6a2c1d79e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52756/0 (socket says 172.21.15.71:52756) 2024-03-25T19:44:59.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.359+0000 7f6a2affd640 1 -- 172.21.15.71:0/4114744326 learned_addr learned my addr 172.21.15.71:0/4114744326 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:44:59.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.359+0000 7f6a13fff640 10 client.?.objecter ms_handle_connect 0x7f6a2c13f710 2024-03-25T19:44:59.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.359+0000 7f6a13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:59.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.363+0000 7f6a13fff640 10 client.?.objecter ms_handle_connect 0x7f6a2c0a98f0 2024-03-25T19:44:59.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.363+0000 7f6a13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:59.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145434967 0 0) 0x7f6a2c1d41e0 con 0x7f6a2c13f710 2024-03-25T19:44:59.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:44:59.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a18004eb0 con 0x7f6a2c13f710 2024-03-25T19:44:59.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715151369 0 0) 0x7f6a2c14e950 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:44:59.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.378+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c1d41e0 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.380+0000 7f6a13fff640 10 client.?.objecter ms_handle_connect 0x7f6a2c1430c0 2024-03-25T19:44:59.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.380+0000 7f6a13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:59.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.386+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465796632 0 0) 0x7f6a2c139840 con 0x7f6a2c1430c0 2024-03-25T19:44:59.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.386+0000 7f6a13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.386+0000 7f6a13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.386+0000 7f6a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.386+0000 7f6a13fff640 10 monclient(hunting): my global_id is 14426 2024-03-25T19:44:59.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.386+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c14e950 con 0x7f6a2c1430c0 2024-03-25T19:44:59.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.402+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2980832615 0 0) 0x7f6a2c14e950 con 0x7f6a2c1430c0 2024-03-25T19:44:59.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.402+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a2c139840 con 0x7f6a2c1430c0 2024-03-25T19:44:59.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.402+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a24001bd0 con 0x7f6a2c1430c0 2024-03-25T19:44:59.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.402+0000 7f6a13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:59.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 10 monclient(hunting): dump: 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3266005890 0 0) 0x7f6a18004eb0 con 0x7f6a2c13f710 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a2c14e950 con 0x7f6a2c13f710 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a20001a30 con 0x7f6a2c13f710 2024-03-25T19:44:59.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.403+0000 7f6a13fff640 10 monclient(hunting): dump: 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:59.405 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.410+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3472551367 0 0) 0x7f6a2c1d41e0 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.410+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a18004eb0 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.410+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a1c001b00 con 0x7f6a2c0a98f0 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.410+0000 7f6a13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.410+0000 7f6a13fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.410+0000 7f6a13fff640 10 monclient(hunting): dump: 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:59.410 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.411 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.411 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.411 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:59.411 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:59.411 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:59.411 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:59.411 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:59.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4250090101 0 0) 0x7f6a2c14e950 con 0x7f6a2c13f710 2024-03-25T19:44:59.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 >> v1:172.21.15.71:6790/0 conn(0x7f6a2c1430c0 legacy=0x7f6a2c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 >> v1:172.21.15.193:6789/0 conn(0x7f6a2c0a98f0 legacy=0x7f6a2c0b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:59.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 1 monclient: found mon.a 2024-03-25T19:44:59.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:59.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c1399d0 con 0x7f6a2c13f710 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 5 monclient: authenticate success, global_id 4419 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 10 monclient: _renew_subs 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2c0b71d0 con 0x7f6a2c13f710 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 10 monclient: _renew_subs 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2c1d82d0 con 0x7f6a2c13f710 2024-03-25T19:44:59.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.420+0000 7f6a3216c640 1 librados: init done 2024-03-25T19:44:59.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.421+0000 7f6a3216c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:59.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.421+0000 7f6a3216c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:59.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.421+0000 7f6a3216c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:59.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.421+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2c13d010 con 0x7f6a2c13f710 2024-03-25T19:44:59.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.740+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a20001c70 con 0x7f6a2c13f710 2024-03-25T19:44:59.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.740+0000 7f6a13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:59.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.740+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6a20003a60 con 0x7f6a2c13f710 2024-03-25T19:44:59.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.740+0000 7f6a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.740+0000 7f6a13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:59.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.740+0000 7f6a13fff640 10 monclient: dump: 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:44:59.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.740+0000 7f6a29ffb640 4 set_mon_vals no callback set 2024-03-25T19:44:59.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.741+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f6a20003c40 con 0x7f6a2c13f710 2024-03-25T19:44:59.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.742+0000 7f6a2b7fe640 10 monclient: get_auth_request con 0x7f6a1806e160 auth_method 0 2024-03-25T19:44:59.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.742+0000 7f6a13fff640 10 client.4419.objecter ms_handle_connect 0x7f6a1806e160 2024-03-25T19:44:59.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.748+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f6a200548e0 con 0x7f6a2c13f710 2024-03-25T19:44:59.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.748+0000 7f6a13fff640 10 client.4419.objecter ms_dispatch 0x7f6a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:59.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.748+0000 7f6a13fff640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:59.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.748+0000 7f6a13fff640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:59.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.748+0000 7f6a13fff640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-25T19:44:59.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.748+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a20050220 con 0x7f6a2c13f710 2024-03-25T19:44:59.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.748+0000 7f6a13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:59.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.748+0000 7f6a13fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:59.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.980+0000 7f6a3216c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:59.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.980+0000 7f6a3216c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:59.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.980+0000 7f6a3216c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:44:59.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:44:59.980+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a2c0b7570 con 0x7f6a2c13f710 2024-03-25T19:45:00.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.053+0000 7f6a13fff640 1 -- 172.21.15.71:0/4114744326 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f6a20052320 con 0x7f6a2c13f710 2024-03-25T19:45:00.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.053+0000 7f6a13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:00.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.053+0000 7f6a13fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:00.054 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:45:00.054 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.055+0000 7f6a3216c640 10 librados: watch_flush enter 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 10 librados: watch_flush exit 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 10 client.4419.objecter successfully canceled tick 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 >> v1:172.21.15.71:6800/3911735624 conn(0x7f6a1806e160 legacy=0x7f6a18070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 10 monclient: shutdown 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 >> v1:172.21.15.71:6789/0 conn(0x7f6a2c13f710 legacy=0x7f6a2c1d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 shutdown_connections 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 >> 172.21.15.71:0/4114744326 conn(0x7f6a2c0b7ea0 msgr2=0x7f6a2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:00.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 shutdown_connections 2024-03-25T19:45:00.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 1 -- 172.21.15.71:0/4114744326 wait complete. 2024-03-25T19:45:00.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:00.056+0000 7f6a3216c640 1 librados: shutdown 2024-03-25T19:45:00.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:02.019 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:04:44.775 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.025 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:04:44.781 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.069 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:04:44.825 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.070 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:44.790 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.079 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:44.799 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.082 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:04:44.838 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.090 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:04:44.846 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.106 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:44.825 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:02.127 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:04:44.883 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:06.874 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:06.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.984+0000 7f2d84a0e640 10 monclient: get_monmap_and_config 2024-03-25T19:45:06.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.984+0000 7f2d84a0e640 10 monclient: build_initial_monmap 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.984+0000 7f2d84a0e640 10 monclient: monmap: 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:45:06.985855+0000 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:45:06.985855+0000 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:45:06.985 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.985+0000 7f2d84a0e640 1 Processor -- start 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.985+0000 7f2d84a0e640 1 -- start start 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.985+0000 7f2d84a0e640 10 monclient: init 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d800b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d800b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d8014be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d800af940 con 0x7f2d8014be90 2024-03-25T19:45:06.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d800b0a70 con 0x7f2d800b4d60 2024-03-25T19:45:06.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d800b0d50 con 0x7f2d800b1730 2024-03-25T19:45:06.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:06.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:06.986+0000 7f2d84a0e640 10 monclient(hunting): authenticate will time out at 5113.298754s 2024-03-25T19:45:07.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.002+0000 7f2d7e59c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f2d800b4d60 0x7f2d8013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:38510/0 (socket says 172.21.15.71:38510) 2024-03-25T19:45:07.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.003+0000 7f2d7e59c640 1 -- 172.21.15.71:0/3857726663 learned_addr learned my addr 172.21.15.71:0/3857726663 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:07.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.020+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610661598 0 0) 0x7f2d800af940 con 0x7f2d8014be90 2024-03-25T19:45:07.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.020+0000 7f2d7dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:07.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.020+0000 7f2d7dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:07.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.020+0000 7f2d7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:07.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.020+0000 7f2d7dd9b640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:45:07.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.020+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d6c003520 con 0x7f2d8014be90 2024-03-25T19:45:07.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.021+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011714164 0 0) 0x7f2d800b0d50 con 0x7f2d800b1730 2024-03-25T19:45:07.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.021+0000 7f2d7dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:07.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.021+0000 7f2d7dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:07.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.021+0000 7f2d7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:07.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.021+0000 7f2d7dd9b640 10 monclient(hunting): my global_id is 14429 2024-03-25T19:45:07.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.021+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d800af940 con 0x7f2d800b1730 2024-03-25T19:45:07.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.022+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530659537 0 0) 0x7f2d800b0a70 con 0x7f2d800b4d60 2024-03-25T19:45:07.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.022+0000 7f2d7dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:07.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.022+0000 7f2d7dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:07.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.022+0000 7f2d7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:07.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.022+0000 7f2d7dd9b640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:45:07.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.022+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d800b0d50 con 0x7f2d800b4d60 2024-03-25T19:45:07.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960089508 0 0) 0x7f2d6c003520 con 0x7f2d8014be90 2024-03-25T19:45:07.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 >> v1:172.21.15.71:6790/0 conn(0x7f2d800b1730 legacy=0x7f2d800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 >> v1:172.21.15.193:6789/0 conn(0x7f2d800b4d60 legacy=0x7f2d8013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:45:07.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d800b1030 con 0x7f2d8014be90 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2d78001a10 con 0x7f2d8014be90 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d7dd9b640 10 monclient: dump: 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.036 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.037 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:07.037 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:07.037 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:07.037 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:07.037 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:07.037 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:07.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d84a0e640 5 monclient: authenticate success, global_id 4422 2024-03-25T19:45:07.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.035+0000 7f2d84a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:07.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.045+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d78001b70 con 0x7f2d8014be90 2024-03-25T19:45:07.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.045+0000 7f2d7dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:07.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.045+0000 7f2d7dd9b640 1 -- 172.21.15.71:0/3857726663 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2d78001f80 con 0x7f2d8014be90 2024-03-25T19:45:07.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.045+0000 7f2d7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:07.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.045+0000 7f2d7dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:07.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.045+0000 7f2d7dd9b640 10 monclient: dump: 2024-03-25T19:45:07.046 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:07.046 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.046+0000 7f2d84a0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:07.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.046+0000 7f2d84a0e640 4 set_mon_vals no callback set 2024-03-25T19:45:07.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.046+0000 7f2d84a0e640 10 monclient: shutdown 2024-03-25T19:45:07.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.046+0000 7f2d84a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:07.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.046+0000 7f2d84a0e640 1 -- 172.21.15.71:0/3857726663 >> v1:172.21.15.71:6789/0 conn(0x7f2d8014be90 legacy=0x7f2d8014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.047+0000 7f2d84a0e640 1 -- 172.21.15.71:0/3857726663 shutdown_connections 2024-03-25T19:45:07.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.047+0000 7f2d84a0e640 1 -- 172.21.15.71:0/3857726663 >> 172.21.15.71:0/3857726663 conn(0x7f2d800617a0 msgr2=0x7f2d80061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.047+0000 7f2d84a0e640 1 -- 172.21.15.71:0/3857726663 shutdown_connections 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.047+0000 7f2d84a0e640 1 -- 172.21.15.71:0/3857726663 wait complete. 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 10 monclient: build_initial_monmap 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 10 monclient: monmap: 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:45:07.050 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 1 librados: starting msgr at 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 1 librados: starting objecter 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 1 Processor -- start 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 1 -- start start 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 1 librados: setting wanted keys 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 1 librados: calling monclient init 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.048+0000 7f2d84a0e640 10 monclient: init 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d800b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d800b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d8014be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d80134a00 con 0x7f2d800b4d60 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d80139050 con 0x7f2d8014be90 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8013a330 con 0x7f2d800b1730 2024-03-25T19:45:07.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:07.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.049+0000 7f2d84a0e640 10 monclient(hunting): authenticate will time out at 5113.361602s 2024-03-25T19:45:07.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.069+0000 7f2d77fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f2d800b4d60 0x7f2d80133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52170/0 (socket says 172.21.15.71:52170) 2024-03-25T19:45:07.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.069+0000 7f2d77fff640 1 -- 172.21.15.71:0/2392434396 learned_addr learned my addr 172.21.15.71:0/2392434396 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:07.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.070+0000 7f2d76ffd640 10 client.?.objecter ms_handle_connect 0x7f2d800b4d60 2024-03-25T19:45:07.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.070+0000 7f2d76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:07.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.073+0000 7f2d76ffd640 10 client.?.objecter ms_handle_connect 0x7f2d800b1730 2024-03-25T19:45:07.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.073+0000 7f2d76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:07.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.077+0000 7f2d76ffd640 10 client.?.objecter ms_handle_connect 0x7f2d8014be90 2024-03-25T19:45:07.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.077+0000 7f2d76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:07.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.079+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385634070 0 0) 0x7f2d8013a330 con 0x7f2d800b1730 2024-03-25T19:45:07.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.079+0000 7f2d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:07.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.079+0000 7f2d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:07.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.079+0000 7f2d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:07.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.079+0000 7f2d76ffd640 10 monclient(hunting): my global_id is 14432 2024-03-25T19:45:07.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.079+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d60004eb0 con 0x7f2d800b1730 2024-03-25T19:45:07.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.080+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343419284 0 0) 0x7f2d80134a00 con 0x7f2d800b4d60 2024-03-25T19:45:07.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.080+0000 7f2d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:07.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.080+0000 7f2d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:07.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.080+0000 7f2d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:07.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.080+0000 7f2d76ffd640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:45:07.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.080+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d8013a330 con 0x7f2d800b4d60 2024-03-25T19:45:07.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.089+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355674359 0 0) 0x7f2d80139050 con 0x7f2d8014be90 2024-03-25T19:45:07.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.089+0000 7f2d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:07.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.089+0000 7f2d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:07.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.089+0000 7f2d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:07.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.089+0000 7f2d76ffd640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:45:07.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.089+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d80134a00 con 0x7f2d8014be90 2024-03-25T19:45:07.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.094+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 514062627 0 0) 0x7f2d60004eb0 con 0x7f2d800b1730 2024-03-25T19:45:07.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.094+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d80139050 con 0x7f2d800b1730 2024-03-25T19:45:07.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.094+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2d700018f0 con 0x7f2d800b1730 2024-03-25T19:45:07.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.094+0000 7f2d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:07.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.094+0000 7f2d76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:07.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.094+0000 7f2d76ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:07.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.094+0000 7f2d76ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:07.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.106+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3706676797 0 0) 0x7f2d8013a330 con 0x7f2d800b4d60 2024-03-25T19:45:07.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.106+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d60004eb0 con 0x7f2d800b4d60 2024-03-25T19:45:07.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.106+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2d680019b0 con 0x7f2d800b4d60 2024-03-25T19:45:07.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.106+0000 7f2d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:07.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.106+0000 7f2d76ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.106+0000 7f2d76ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:07.107 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:07.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.112+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3057504548 0 0) 0x7f2d80134a00 con 0x7f2d8014be90 2024-03-25T19:45:07.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.112+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d8013a330 con 0x7f2d8014be90 2024-03-25T19:45:07.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.112+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2d78001e80 con 0x7f2d8014be90 2024-03-25T19:45:07.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.112+0000 7f2d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:07.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.112+0000 7f2d76ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:07.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.112+0000 7f2d76ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:07.113 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:07.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1546995103 0 0) 0x7f2d60004eb0 con 0x7f2d800b4d60 2024-03-25T19:45:07.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 >> v1:172.21.15.71:6790/0 conn(0x7f2d800b1730 legacy=0x7f2d8013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 >> v1:172.21.15.193:6789/0 conn(0x7f2d8014be90 legacy=0x7f2d801342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:07.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 1 monclient: found mon.a 2024-03-25T19:45:07.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:07.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8013a610 con 0x7f2d800b4d60 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 5 monclient: authenticate success, global_id 4425 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 10 monclient: _renew_subs 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8013a8f0 con 0x7f2d800b4d60 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 10 monclient: _renew_subs 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d80139730 con 0x7f2d800b4d60 2024-03-25T19:45:07.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.116+0000 7f2d84a0e640 1 librados: init done 2024-03-25T19:45:07.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.117+0000 7f2d84a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:07.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.117+0000 7f2d84a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:07.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.117+0000 7f2d84a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:07.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.117+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d801391e0 con 0x7f2d800b4d60 2024-03-25T19:45:07.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.433+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d680035f0 con 0x7f2d800b4d60 2024-03-25T19:45:07.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.433+0000 7f2d76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:07.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.433+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2d680037e0 con 0x7f2d800b4d60 2024-03-25T19:45:07.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.433+0000 7f2d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.433+0000 7f2d76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.433+0000 7f2d76ffd640 10 monclient: dump: 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:07.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.433+0000 7f2d7d59a640 4 set_mon_vals no callback set 2024-03-25T19:45:07.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.434+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f2d68003a00 con 0x7f2d800b4d60 2024-03-25T19:45:07.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.435+0000 7f2d7e59c640 10 monclient: get_auth_request con 0x7f2d6006e160 auth_method 0 2024-03-25T19:45:07.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.435+0000 7f2d76ffd640 10 client.4425.objecter ms_handle_connect 0x7f2d6006e160 2024-03-25T19:45:07.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.440+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f2d68054610 con 0x7f2d800b4d60 2024-03-25T19:45:07.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.440+0000 7f2d76ffd640 10 client.4425.objecter ms_dispatch 0x7f2d80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:07.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.440+0000 7f2d76ffd640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:07.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.440+0000 7f2d76ffd640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:07.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.440+0000 7f2d76ffd640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-25T19:45:07.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.440+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d68050120 con 0x7f2d800b4d60 2024-03-25T19:45:07.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.440+0000 7f2d76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:07.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.440+0000 7f2d76ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:07.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.685+0000 7f2d84a0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:07.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.685+0000 7f2d84a0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:07.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.685+0000 7f2d84a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:07.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.685+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d800627b0 con 0x7f2d800b4d60 2024-03-25T19:45:07.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.729+0000 7f2d76ffd640 1 -- 172.21.15.71:0/2392434396 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f2d68052050 con 0x7f2d800b4d60 2024-03-25T19:45:07.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.729+0000 7f2d76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:07.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.729+0000 7f2d76ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:07.729 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:45:07.729 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:07.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 10 librados: watch_flush enter 2024-03-25T19:45:07.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 10 librados: watch_flush exit 2024-03-25T19:45:07.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-25T19:45:07.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 10 client.4425.objecter successfully canceled tick 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 >> v1:172.21.15.71:6800/3911735624 conn(0x7f2d6006e160 legacy=0x7f2d60070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 10 monclient: shutdown 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 >> v1:172.21.15.71:6789/0 conn(0x7f2d800b4d60 legacy=0x7f2d80133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 shutdown_connections 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 >> 172.21.15.71:0/2392434396 conn(0x7f2d800617a0 msgr2=0x7f2d800b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 shutdown_connections 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 1 -- 172.21.15.71:0/2392434396 wait complete. 2024-03-25T19:45:07.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:07.731+0000 7f2d84a0e640 1 librados: shutdown 2024-03-25T19:45:07.743 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:14.549 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:14.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.663+0000 7fec7424a640 10 monclient: get_monmap_and_config 2024-03-25T19:45:14.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.663+0000 7fec7424a640 10 monclient: build_initial_monmap 2024-03-25T19:45:14.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.663+0000 7fec7424a640 10 monclient: monmap: 2024-03-25T19:45:14.663 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:14.663 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:14.664 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:45:14.664670+0000 2024-03-25T19:45:14.664 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:45:14.664670+0000 2024-03-25T19:45:14.664 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:14.664 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:14.664 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:14.664 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:45:14.664 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:45:14.664 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:14.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.664+0000 7fec7424a640 1 Processor -- start 2024-03-25T19:45:14.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.664+0000 7fec7424a640 1 -- start start 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.664+0000 7fec7424a640 10 monclient: init 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 10 monclient(hunting): picked mon.noname-b con 0x7fec6c0b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 10 monclient(hunting): picked mon.noname-a con 0x7fec6c0b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 10 monclient(hunting): picked mon.noname-c con 0x7fec6c14be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6c0af940 con 0x7fec6c0b1730 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6c0b0a70 con 0x7fec6c14be90 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6c0b0d50 con 0x7fec6c0b4d60 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:14.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.665+0000 7fec7424a640 10 monclient(hunting): authenticate will time out at 5120.977291s 2024-03-25T19:45:14.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.670+0000 7fec727e7640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fec6c14be90 0x7fec6c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:43496/0 (socket says 172.21.15.71:43496) 2024-03-25T19:45:14.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.670+0000 7fec727e7640 1 -- 172.21.15.71:0/4093337167 learned_addr learned my addr 172.21.15.71:0/4093337167 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:14.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.706+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480831111 0 0) 0x7fec6c0af940 con 0x7fec6c0b1730 2024-03-25T19:45:14.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.706+0000 7fec70fe4640 10 monclient(hunting): _negotiate 2024-03-25T19:45:14.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.706+0000 7fec70fe4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:14.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.706+0000 7fec70fe4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:14.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.706+0000 7fec70fe4640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:45:14.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.706+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec58003520 con 0x7fec6c0b1730 2024-03-25T19:45:14.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4138950813 0 0) 0x7fec58003520 con 0x7fec6c0b1730 2024-03-25T19:45:14.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 >> v1:172.21.15.71:6790/0 conn(0x7fec6c0b4d60 legacy=0x7fec6c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:14.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 >> v1:172.21.15.193:6789/0 conn(0x7fec6c14be90 legacy=0x7fec6c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:14.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 10 monclient: _finish_hunting 0 2024-03-25T19:45:14.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 1 monclient: found mon.noname-a 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c0b1030 con 0x7fec6c0b1730 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 10 monclient: _finish_auth 0 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 10 monclient: _check_auth_tickets 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fec5c001690 con 0x7fec6c0b1730 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec70fe4640 10 monclient: dump: 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:14.765 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:14.766 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:14.766 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:14.766 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:14.766 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:14.766 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:14.766 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:14.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec7424a640 5 monclient: authenticate success, global_id 4428 2024-03-25T19:45:14.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.764+0000 7fec7424a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.789+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec5c0017f0 con 0x7fec6c0b1730 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.789+0000 7fec70fe4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.789+0000 7fec70fe4640 1 -- 172.21.15.71:0/4093337167 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fec5c001c40 con 0x7fec6c0b1730 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.789+0000 7fec7424a640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.789+0000 7fec7424a640 4 set_mon_vals no callback set 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.789+0000 7fec70fe4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.789+0000 7fec70fe4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.789+0000 7fec70fe4640 10 monclient: dump: 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:14.789 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:14.790 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:14.790 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:14.790 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:14.790 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:14.790 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:14.790 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:14.790 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:14.790 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.790+0000 7fec7424a640 10 monclient: shutdown 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.790+0000 7fec7424a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.790+0000 7fec7424a640 1 -- 172.21.15.71:0/4093337167 >> v1:172.21.15.71:6789/0 conn(0x7fec6c0b1730 legacy=0x7fec6c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.790+0000 7fec7424a640 1 -- 172.21.15.71:0/4093337167 shutdown_connections 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.790+0000 7fec7424a640 1 -- 172.21.15.71:0/4093337167 >> 172.21.15.71:0/4093337167 conn(0x7fec6c0617a0 msgr2=0x7fec6c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.790+0000 7fec7424a640 1 -- 172.21.15.71:0/4093337167 shutdown_connections 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.790+0000 7fec7424a640 1 -- 172.21.15.71:0/4093337167 wait complete. 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 10 monclient: build_initial_monmap 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 10 monclient: monmap: 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:14.792 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 1 librados: starting msgr at 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 1 librados: starting objecter 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 1 Processor -- start 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 1 -- start start 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 1 librados: setting wanted keys 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 1 librados: calling monclient init 2024-03-25T19:45:14.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.791+0000 7fec7424a640 10 monclient: init 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 10 monclient(hunting): picked mon.noname-b con 0x7fec6c0b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 10 monclient(hunting): picked mon.noname-c con 0x7fec6c0b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 10 monclient(hunting): picked mon.noname-a con 0x7fec6c14be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6c1420d0 con 0x7fec6c14be90 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6c138270 con 0x7fec6c0b1730 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6c138400 con 0x7fec6c0b4d60 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:14.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.792+0000 7fec7424a640 10 monclient(hunting): authenticate will time out at 5121.104537s 2024-03-25T19:45:14.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.805+0000 7fec727e7640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fec6c14be90 0x7fec6c134d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:56994/0 (socket says 172.21.15.71:56994) 2024-03-25T19:45:14.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.805+0000 7fec727e7640 1 -- 172.21.15.71:0/3939031289 learned_addr learned my addr 172.21.15.71:0/3939031289 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:14.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.808+0000 7fec627fc640 10 client.?.objecter ms_handle_connect 0x7fec6c14be90 2024-03-25T19:45:14.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.808+0000 7fec627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:14.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.824+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434741429 0 0) 0x7fec6c1420d0 con 0x7fec6c14be90 2024-03-25T19:45:14.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.824+0000 7fec627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:14.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.824+0000 7fec627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:14.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.824+0000 7fec627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:14.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.824+0000 7fec627fc640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:45:14.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.824+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec50004eb0 con 0x7fec6c14be90 2024-03-25T19:45:14.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.839+0000 7fec627fc640 10 client.?.objecter ms_handle_connect 0x7fec6c0b1730 2024-03-25T19:45:14.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.839+0000 7fec627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.840+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518266814 0 0) 0x7fec50004eb0 con 0x7fec6c14be90 2024-03-25T19:45:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.840+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec6c1420d0 con 0x7fec6c14be90 2024-03-25T19:45:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.840+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fec64001b10 con 0x7fec6c14be90 2024-03-25T19:45:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.840+0000 7fec627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.840+0000 7fec627fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.840+0000 7fec627fc640 10 monclient(hunting): dump: 2024-03-25T19:45:14.840 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:14.840 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:14.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.840+0000 7fec627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.852+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291981398 0 0) 0x7fec6c138270 con 0x7fec6c0b1730 2024-03-25T19:45:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.852+0000 7fec627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.852+0000 7fec627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.852+0000 7fec627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.852+0000 7fec627fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:45:14.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.852+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec50004eb0 con 0x7fec6c0b1730 2024-03-25T19:45:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.859+0000 7fec627fc640 10 client.?.objecter ms_handle_connect 0x7fec6c0b4d60 2024-03-25T19:45:14.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.859+0000 7fec627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:14.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.866+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955427338 0 0) 0x7fec6c138400 con 0x7fec6c0b4d60 2024-03-25T19:45:14.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.867+0000 7fec627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:14.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.867+0000 7fec627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:14.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.867+0000 7fec627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:14.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.867+0000 7fec627fc640 10 monclient(hunting): my global_id is 14435 2024-03-25T19:45:14.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.867+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec6c138270 con 0x7fec6c0b4d60 2024-03-25T19:45:14.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3688927054 0 0) 0x7fec6c1420d0 con 0x7fec6c14be90 2024-03-25T19:45:14.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 >> v1:172.21.15.71:6790/0 conn(0x7fec6c0b4d60 legacy=0x7fec6c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:14.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 >> v1:172.21.15.193:6789/0 conn(0x7fec6c0b1730 legacy=0x7fec6c1db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:14.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:14.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 1 monclient: found mon.a 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c142480 con 0x7fec6c14be90 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 5 monclient: authenticate success, global_id 4431 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 10 monclient: _renew_subs 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6c1426d0 con 0x7fec6c14be90 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 10 monclient: _renew_subs 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6c1dba90 con 0x7fec6c14be90 2024-03-25T19:45:14.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.870+0000 7fec7424a640 1 librados: init done 2024-03-25T19:45:14.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.871+0000 7fec7424a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:14.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.871+0000 7fec7424a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:14.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.871+0000 7fec7424a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:14.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:14.871+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6c1401f0 con 0x7fec6c14be90 2024-03-25T19:45:15.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.216+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec64002080 con 0x7fec6c14be90 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.216+0000 7fec627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.216+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fec64004d00 con 0x7fec6c14be90 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.216+0000 7fec627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.216+0000 7fec627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.216+0000 7fec627fc640 10 monclient: dump: 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:15.216 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:15.217 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:15.217 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:15.217 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:15.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.216+0000 7fec63fff640 4 set_mon_vals no callback set 2024-03-25T19:45:15.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.217+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fec64004f20 con 0x7fec6c14be90 2024-03-25T19:45:15.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.217+0000 7fec71fe6640 10 monclient: get_auth_request con 0x7fec5006de10 auth_method 0 2024-03-25T19:45:15.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.218+0000 7fec627fc640 10 client.4431.objecter ms_handle_connect 0x7fec5006de10 2024-03-25T19:45:15.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.221+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fec64054a50 con 0x7fec6c14be90 2024-03-25T19:45:15.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.221+0000 7fec627fc640 10 client.4431.objecter ms_dispatch 0x7fec6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:15.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.221+0000 7fec627fc640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:15.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.221+0000 7fec627fc640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:15.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.221+0000 7fec627fc640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-25T19:45:15.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.221+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec64050390 con 0x7fec6c14be90 2024-03-25T19:45:15.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.221+0000 7fec627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:15.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.221+0000 7fec627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:15.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.472+0000 7fec7424a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:15.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.472+0000 7fec7424a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:15.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.472+0000 7fec7424a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:15.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.472+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec6c0627b0 con 0x7fec6c14be90 2024-03-25T19:45:15.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.497+0000 7fec627fc640 1 -- 172.21.15.71:0/3939031289 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fec64052490 con 0x7fec6c14be90 2024-03-25T19:45:15.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.497+0000 7fec627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:15.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.497+0000 7fec627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:15.498 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:45:15.498 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.499+0000 7fec7424a640 10 librados: watch_flush enter 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 10 librados: watch_flush exit 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 10 client.4431.objecter successfully canceled tick 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 >> v1:172.21.15.71:6800/3911735624 conn(0x7fec5006de10 legacy=0x7fec500702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 10 monclient: shutdown 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 >> v1:172.21.15.71:6789/0 conn(0x7fec6c14be90 legacy=0x7fec6c134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 shutdown_connections 2024-03-25T19:45:15.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 >> 172.21.15.71:0/3939031289 conn(0x7fec6c0617a0 msgr2=0x7fec6c0b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:15.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 shutdown_connections 2024-03-25T19:45:15.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 1 -- 172.21.15.71:0/3939031289 wait complete. 2024-03-25T19:45:15.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:15.500+0000 7fec7424a640 1 librados: shutdown 2024-03-25T19:45:15.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:22.065 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:04.821 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.072 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:04.828 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.101 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:04.857 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.127 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:04.847 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.139 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:04.858 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.147 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:04.903 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.153 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:04.909 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.164 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:04.883 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.182 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:04.938 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:22.316 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:22.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.429+0000 7ff6e8b65640 10 monclient: get_monmap_and_config 2024-03-25T19:45:22.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.429+0000 7ff6e8b65640 10 monclient: build_initial_monmap 2024-03-25T19:45:22.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.429+0000 7ff6e8b65640 10 monclient: monmap: 2024-03-25T19:45:22.429 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:22.429 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:22.429 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:45:22.430625+0000 2024-03-25T19:45:22.430 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:45:22.430625+0000 2024-03-25T19:45:22.430 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:22.430 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:22.430 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:22.430 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:45:22.430 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:45:22.430 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:22.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.430+0000 7ff6e8b65640 1 Processor -- start 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.430+0000 7ff6e8b65640 1 -- start start 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.430+0000 7ff6e8b65640 10 monclient: init 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e40b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e40b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e414be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e40af940 con 0x7ff6e40b4d60 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e40b0a70 con 0x7ff6e414be90 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e40b0d50 con 0x7ff6e40b1730 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:22.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.431+0000 7ff6e8b65640 10 monclient(hunting): authenticate will time out at 5128.743806s 2024-03-25T19:45:22.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.439+0000 7ff6e2d9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff6e414be90 0x7ff6e414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:54170/0 (socket says 172.21.15.71:54170) 2024-03-25T19:45:22.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.440+0000 7ff6e2d9d640 1 -- 172.21.15.71:0/2726523854 learned_addr learned my addr 172.21.15.71:0/2726523854 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:22.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.449+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632719207 0 0) 0x7ff6e40b0a70 con 0x7ff6e414be90 2024-03-25T19:45:22.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.449+0000 7ff6e159a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.449+0000 7ff6e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.449+0000 7ff6e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.449+0000 7ff6e159a640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:45:22.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.449+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6cc003520 con 0x7ff6e414be90 2024-03-25T19:45:22.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.455+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914611886 0 0) 0x7ff6e40b0d50 con 0x7ff6e40b1730 2024-03-25T19:45:22.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.455+0000 7ff6e159a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.455+0000 7ff6e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.455+0000 7ff6e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.455+0000 7ff6e159a640 10 monclient(hunting): my global_id is 14438 2024-03-25T19:45:22.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.455+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e40b0a70 con 0x7ff6e40b1730 2024-03-25T19:45:22.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738659755 0 0) 0x7ff6cc003520 con 0x7ff6e414be90 2024-03-25T19:45:22.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 >> v1:172.21.15.71:6790/0 conn(0x7ff6e40b1730 legacy=0x7ff6e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 >> v1:172.21.15.71:6789/0 conn(0x7ff6e40b4d60 legacy=0x7ff6e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 10 monclient: _finish_hunting 0 2024-03-25T19:45:22.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 1 monclient: found mon.noname-c 2024-03-25T19:45:22.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e40b1030 con 0x7ff6e414be90 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 10 monclient: _finish_auth 0 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 10 monclient: _check_auth_tickets 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff6dc001a10 con 0x7ff6e414be90 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e159a640 10 monclient: dump: 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:22.459 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:22.460 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:22.460 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:22.460 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:22.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e8b65640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:45:22.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.458+0000 7ff6e8b65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:22.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.463+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6dc001b70 con 0x7ff6e414be90 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.463+0000 7ff6e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.463+0000 7ff6e159a640 1 -- 172.21.15.71:0/2726523854 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff6dc001f80 con 0x7ff6e414be90 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.463+0000 7ff6e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.463+0000 7ff6e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.463+0000 7ff6e159a640 10 monclient: dump: 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:22.463 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:22.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.463+0000 7ff6e8b65640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:22.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.463+0000 7ff6e8b65640 4 set_mon_vals no callback set 2024-03-25T19:45:22.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.464+0000 7ff6e8b65640 10 monclient: shutdown 2024-03-25T19:45:22.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.464+0000 7ff6e8b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:22.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.464+0000 7ff6e8b65640 1 -- 172.21.15.71:0/2726523854 >> v1:172.21.15.193:6789/0 conn(0x7ff6e414be90 legacy=0x7ff6e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.464+0000 7ff6e8b65640 1 -- 172.21.15.71:0/2726523854 shutdown_connections 2024-03-25T19:45:22.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.464+0000 7ff6e8b65640 1 -- 172.21.15.71:0/2726523854 >> 172.21.15.71:0/2726523854 conn(0x7ff6e40617a0 msgr2=0x7ff6e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:22.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.464+0000 7ff6e8b65640 1 -- 172.21.15.71:0/2726523854 shutdown_connections 2024-03-25T19:45:22.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.464+0000 7ff6e8b65640 1 -- 172.21.15.71:0/2726523854 wait complete. 2024-03-25T19:45:22.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.465+0000 7ff6e8b65640 10 monclient: build_initial_monmap 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.465+0000 7ff6e8b65640 10 monclient: monmap: 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 1 librados: starting msgr at 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 1 librados: starting objecter 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 1 Processor -- start 2024-03-25T19:45:22.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 1 -- start start 2024-03-25T19:45:22.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 1 librados: setting wanted keys 2024-03-25T19:45:22.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 1 librados: calling monclient init 2024-03-25T19:45:22.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 10 monclient: init 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e40b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e40b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e414be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.466+0000 7ff6e8b65640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e4135ee0 con 0x7ff6e414be90 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.467+0000 7ff6e8b65640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e4135b00 con 0x7ff6e40b4d60 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.467+0000 7ff6e8b65640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e4135c90 con 0x7ff6e40b1730 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.467+0000 7ff6e8b65640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:22.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.467+0000 7ff6e8b65640 10 monclient(hunting): authenticate will time out at 5128.778897s 2024-03-25T19:45:22.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.483+0000 7ff6e2d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff6e414be90 0x7ff6e41d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53820/0 (socket says 172.21.15.71:53820) 2024-03-25T19:45:22.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.483+0000 7ff6e2d9d640 1 -- 172.21.15.71:0/3951442728 learned_addr learned my addr 172.21.15.71:0/3951442728 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:22.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.488+0000 7ff6d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff6e414be90 2024-03-25T19:45:22.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.488+0000 7ff6d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:22.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.488+0000 7ff6d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff6e40b4d60 2024-03-25T19:45:22.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.488+0000 7ff6d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:22.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.494+0000 7ff6d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff6e40b1730 2024-03-25T19:45:22.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.494+0000 7ff6d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:22.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.497+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154844566 0 0) 0x7ff6e4135b00 con 0x7ff6e40b4d60 2024-03-25T19:45:22.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.497+0000 7ff6d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.497+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.497+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.497+0000 7ff6d2ffd640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:45:22.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.497+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c4004eb0 con 0x7ff6e40b4d60 2024-03-25T19:45:22.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.502+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766208971 0 0) 0x7ff6e4135c90 con 0x7ff6e40b1730 2024-03-25T19:45:22.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.502+0000 7ff6d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.502+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.502+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.503+0000 7ff6d2ffd640 10 monclient(hunting): my global_id is 14441 2024-03-25T19:45:22.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.503+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e4135b00 con 0x7ff6e40b1730 2024-03-25T19:45:22.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.506+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632188472 0 0) 0x7ff6e4135ee0 con 0x7ff6e414be90 2024-03-25T19:45:22.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.506+0000 7ff6d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.506+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.506+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.506+0000 7ff6d2ffd640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:45:22.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.506+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e4135c90 con 0x7ff6e414be90 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.510+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1017346407 0 0) 0x7ff6c4004eb0 con 0x7ff6e40b4d60 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.510+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e4135ee0 con 0x7ff6e40b4d60 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.510+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff6d8001930 con 0x7ff6e40b4d60 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.510+0000 7ff6d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.510+0000 7ff6d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.510+0000 7ff6d2ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:22.510 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:22.511 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.511 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:22.511 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:22.511 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:22.511 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:22.511 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:22.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.510+0000 7ff6d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:22.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2690312512 0 0) 0x7ff6e4135ee0 con 0x7ff6e40b4d60 2024-03-25T19:45:22.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 >> v1:172.21.15.71:6790/0 conn(0x7ff6e40b1730 legacy=0x7ff6e412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 >> v1:172.21.15.71:6789/0 conn(0x7ff6e414be90 legacy=0x7ff6e41d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:22.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 1 monclient: found mon.b 2024-03-25T19:45:22.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4136960 con 0x7ff6e40b4d60 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 5 monclient: authenticate success, global_id 4435 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 10 monclient: _renew_subs 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e4130470 con 0x7ff6e40b4d60 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 10 monclient: _renew_subs 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e413e870 con 0x7ff6e40b4d60 2024-03-25T19:45:22.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.515+0000 7ff6e8b65640 1 librados: init done 2024-03-25T19:45:22.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.516+0000 7ff6e8b65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:22.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.516+0000 7ff6e8b65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:22.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.516+0000 7ff6e8b65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:22.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.516+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e4130810 con 0x7ff6e40b4d60 2024-03-25T19:45:22.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.789+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d8003610 con 0x7ff6e40b4d60 2024-03-25T19:45:22.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.789+0000 7ff6d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.789+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff6d8003880 con 0x7ff6e40b4d60 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.789+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.789+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.789+0000 7ff6d2ffd640 10 monclient: dump: 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.790 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:22.791 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:22.791 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:22.791 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:22.791 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:22.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.789+0000 7ff6e0d99640 4 set_mon_vals no callback set 2024-03-25T19:45:22.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.790+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7ff6d8003aa0 con 0x7ff6e40b4d60 2024-03-25T19:45:22.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.791+0000 7ff6e259c640 10 monclient: get_auth_request con 0x7ff6c406ddf0 auth_method 0 2024-03-25T19:45:22.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.791+0000 7ff6d2ffd640 10 client.4435.objecter ms_handle_connect 0x7ff6c406ddf0 2024-03-25T19:45:22.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.795+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7ff6d8053460 con 0x7ff6e40b4d60 2024-03-25T19:45:22.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.796+0000 7ff6d2ffd640 10 client.4435.objecter ms_dispatch 0x7ff6e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:22.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.796+0000 7ff6d2ffd640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:22.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.796+0000 7ff6d2ffd640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:22.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.796+0000 7ff6d2ffd640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T19:45:22.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.796+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6d804ef70 con 0x7ff6e40b4d60 2024-03-25T19:45:22.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.796+0000 7ff6d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:22.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:22.796+0000 7ff6d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:23.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.052+0000 7ff6e8b65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:23.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.052+0000 7ff6e8b65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:23.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.052+0000 7ff6e8b65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:23.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.052+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6e4130600 con 0x7ff6e40b4d60 2024-03-25T19:45:23.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.207+0000 7ff6d2ffd640 1 -- 172.21.15.71:0/3951442728 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7ff6d8050ea0 con 0x7ff6e40b4d60 2024-03-25T19:45:23.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.207+0000 7ff6d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:23.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.207+0000 7ff6d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:23.207 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:45:23.207 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:23.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 10 librados: watch_flush enter 2024-03-25T19:45:23.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 10 librados: watch_flush exit 2024-03-25T19:45:23.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T19:45:23.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 10 client.4435.objecter successfully canceled tick 2024-03-25T19:45:23.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 >> v1:172.21.15.71:6800/3911735624 conn(0x7ff6c406ddf0 legacy=0x7ff6c40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:23.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 10 monclient: shutdown 2024-03-25T19:45:23.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:23.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 >> v1:172.21.15.193:6789/0 conn(0x7ff6e40b4d60 legacy=0x7ff6e41d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:23.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 shutdown_connections 2024-03-25T19:45:23.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 >> 172.21.15.71:0/3951442728 conn(0x7ff6e40617a0 msgr2=0x7ff6e40b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:23.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 shutdown_connections 2024-03-25T19:45:23.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 1 -- 172.21.15.71:0/3951442728 wait complete. 2024-03-25T19:45:23.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:23.209+0000 7ff6e8b65640 1 librados: shutdown 2024-03-25T19:45:23.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:30.027 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:30.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.141+0000 7fd885455640 10 monclient: get_monmap_and_config 2024-03-25T19:45:30.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.141+0000 7fd885455640 10 monclient: build_initial_monmap 2024-03-25T19:45:30.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.141+0000 7fd885455640 10 monclient: monmap: 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:45:30.142762+0000 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:45:30.142762+0000 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:45:30.142 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:30.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.142+0000 7fd885455640 1 Processor -- start 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.142+0000 7fd885455640 1 -- start start 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.142+0000 7fd885455640 10 monclient: init 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 10 monclient(hunting): picked mon.noname-b con 0x7fd8800a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 10 monclient(hunting): picked mon.noname-c con 0x7fd88013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 10 monclient(hunting): picked mon.noname-a con 0x7fd8801430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd880142520 con 0x7fd8801430c0 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8800b28a0 con 0x7fd88013f710 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8800b2a30 con 0x7fd8800a98f0 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:30.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.143+0000 7fd885455640 10 monclient(hunting): authenticate will time out at 5136.455341s 2024-03-25T19:45:30.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.145+0000 7fd87e7fc640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd88013f710 0x7fd8801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:48408/0 (socket says 172.21.15.71:48408) 2024-03-25T19:45:30.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.145+0000 7fd87e7fc640 1 -- 172.21.15.71:0/3615083865 learned_addr learned my addr 172.21.15.71:0/3615083865 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:30.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.157+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7136186 0 0) 0x7fd8800b28a0 con 0x7fd88013f710 2024-03-25T19:45:30.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.157+0000 7fd87dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:30.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.157+0000 7fd87dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:30.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.157+0000 7fd87dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:30.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.157+0000 7fd87dffb640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:45:30.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.157+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd868003520 con 0x7fd88013f710 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4280507764 0 0) 0x7fd868003520 con 0x7fd88013f710 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 >> v1:172.21.15.71:6790/0 conn(0x7fd8800a98f0 legacy=0x7fd8800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 >> v1:172.21.15.71:6789/0 conn(0x7fd8801430c0 legacy=0x7fd88014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 1 monclient: found mon.noname-c 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8800b44c0 con 0x7fd88013f710 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:30.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd874001690 con 0x7fd88013f710 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd87dffb640 10 monclient: dump: 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd885455640 5 monclient: authenticate success, global_id 4438 2024-03-25T19:45:30.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.162+0000 7fd885455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:30.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.179+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd874001860 con 0x7fd88013f710 2024-03-25T19:45:30.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.179+0000 7fd87dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:30.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.179+0000 7fd87dffb640 1 -- 172.21.15.71:0/3615083865 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd874001c70 con 0x7fd88013f710 2024-03-25T19:45:30.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.179+0000 7fd87dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:30.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.179+0000 7fd87dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:30.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.179+0000 7fd87dffb640 10 monclient: dump: 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.179+0000 7fd885455640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:30.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.179+0000 7fd885455640 4 set_mon_vals no callback set 2024-03-25T19:45:30.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.180+0000 7fd885455640 10 monclient: shutdown 2024-03-25T19:45:30.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.180+0000 7fd885455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:30.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.180+0000 7fd885455640 1 -- 172.21.15.71:0/3615083865 >> v1:172.21.15.193:6789/0 conn(0x7fd88013f710 legacy=0x7fd8801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:30.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.180+0000 7fd885455640 1 -- 172.21.15.71:0/3615083865 shutdown_connections 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.180+0000 7fd885455640 1 -- 172.21.15.71:0/3615083865 >> 172.21.15.71:0/3615083865 conn(0x7fd8800b7ea0 msgr2=0x7fd8800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.180+0000 7fd885455640 1 -- 172.21.15.71:0/3615083865 shutdown_connections 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.180+0000 7fd885455640 1 -- 172.21.15.71:0/3615083865 wait complete. 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.181+0000 7fd885455640 10 monclient: build_initial_monmap 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.181+0000 7fd885455640 10 monclient: monmap: 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:30.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.181+0000 7fd885455640 1 librados: starting msgr at 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.181+0000 7fd885455640 1 librados: starting objecter 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 1 Processor -- start 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 1 -- start start 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 1 librados: setting wanted keys 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 1 librados: calling monclient init 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 10 monclient: init 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 10 monclient(hunting): picked mon.noname-c con 0x7fd8800a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 10 monclient(hunting): picked mon.noname-a con 0x7fd88013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 10 monclient(hunting): picked mon.noname-b con 0x7fd8801430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:30.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd880139840 con 0x7fd88013f710 2024-03-25T19:45:30.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd880138b90 con 0x7fd8801430c0 2024-03-25T19:45:30.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd880138790 con 0x7fd8800a98f0 2024-03-25T19:45:30.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:30.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.182+0000 7fd885455640 10 monclient(hunting): authenticate will time out at 5136.494763s 2024-03-25T19:45:30.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.195+0000 7fd87e7fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd88013f710 0x7fd8801d7790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58964/0 (socket says 172.21.15.71:58964) 2024-03-25T19:45:30.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.195+0000 7fd87e7fc640 1 -- 172.21.15.71:0/662659851 learned_addr learned my addr 172.21.15.71:0/662659851 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:30.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.196+0000 7fd8677fe640 10 client.?.objecter ms_handle_connect 0x7fd88013f710 2024-03-25T19:45:30.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.196+0000 7fd8677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:30.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.200+0000 7fd8677fe640 10 client.?.objecter ms_handle_connect 0x7fd8801430c0 2024-03-25T19:45:30.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.200+0000 7fd8677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:30.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.204+0000 7fd8677fe640 10 client.?.objecter ms_handle_connect 0x7fd8800a98f0 2024-03-25T19:45:30.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.204+0000 7fd8677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:30.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.208+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421261475 0 0) 0x7fd880138b90 con 0x7fd8801430c0 2024-03-25T19:45:30.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.208+0000 7fd8677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:30.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.208+0000 7fd8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:30.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.208+0000 7fd8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:30.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.208+0000 7fd8677fe640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:45:30.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.208+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd86c004eb0 con 0x7fd8801430c0 2024-03-25T19:45:30.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.214+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922219872 0 0) 0x7fd880139840 con 0x7fd88013f710 2024-03-25T19:45:30.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.214+0000 7fd8677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:30.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.214+0000 7fd8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:30.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.214+0000 7fd8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:30.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.214+0000 7fd8677fe640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:45:30.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.214+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd880138b90 con 0x7fd88013f710 2024-03-25T19:45:30.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414309233 0 0) 0x7fd880138790 con 0x7fd8800a98f0 2024-03-25T19:45:30.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:30.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:30.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:30.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 10 monclient(hunting): my global_id is 14444 2024-03-25T19:45:30.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd880139840 con 0x7fd8800a98f0 2024-03-25T19:45:30.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3652843420 0 0) 0x7fd880138b90 con 0x7fd88013f710 2024-03-25T19:45:30.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd880138790 con 0x7fd88013f710 2024-03-25T19:45:30.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd874001b70 con 0x7fd88013f710 2024-03-25T19:45:30.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:30.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 10 monclient(hunting): dump: 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:30.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.231+0000 7fd8677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:30.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1283496921 0 0) 0x7fd880138790 con 0x7fd88013f710 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 >> v1:172.21.15.71:6790/0 conn(0x7fd8800a98f0 legacy=0x7fd880138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 >> v1:172.21.15.193:6789/0 conn(0x7fd8801430c0 legacy=0x7fd8801d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 1 monclient: found mon.a 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd880138920 con 0x7fd88013f710 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:30.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd8677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd885455640 5 monclient: authenticate success, global_id 4443 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd885455640 10 monclient: _renew_subs 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd885455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.246+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800b15a0 con 0x7fd88013f710 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 10 monclient: _renew_subs 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8801e3230 con 0x7fd88013f710 2024-03-25T19:45:30.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 1 librados: init done 2024-03-25T19:45:30.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:30.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:30.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:30.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.247+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8800627b0 con 0x7fd88013f710 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.550+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd874001d10 con 0x7fd88013f710 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.551+0000 7fd8677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.551+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd874004d90 con 0x7fd88013f710 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.551+0000 7fd8677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.551+0000 7fd8677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.551+0000 7fd8677fe640 10 monclient: dump: 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:30.551 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:30.552 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:30.552 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:30.552 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:30.552 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:30.552 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:30.552 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:30.552 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:30.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.551+0000 7fd87d7fa640 4 set_mon_vals no callback set 2024-03-25T19:45:30.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.551+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fd87401ad60 con 0x7fd88013f710 2024-03-25T19:45:30.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.552+0000 7fd87effd640 10 monclient: get_auth_request con 0x7fd86c06ddd0 auth_method 0 2024-03-25T19:45:30.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.552+0000 7fd8677fe640 10 client.4443.objecter ms_handle_connect 0x7fd86c06ddd0 2024-03-25T19:45:30.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.557+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fd874054900 con 0x7fd88013f710 2024-03-25T19:45:30.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.557+0000 7fd8677fe640 10 client.4443.objecter ms_dispatch 0x7fd880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:30.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.557+0000 7fd8677fe640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:30.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.557+0000 7fd8677fe640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:30.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.557+0000 7fd8677fe640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T19:45:30.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.557+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd874050240 con 0x7fd88013f710 2024-03-25T19:45:30.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.557+0000 7fd8677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:30.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.557+0000 7fd8677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.804+0000 7fd885455640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.804+0000 7fd885455640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.804+0000 7fd885455640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.804+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8800b7180 con 0x7fd88013f710 2024-03-25T19:45:30.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.880+0000 7fd8677fe640 1 -- 172.21.15.71:0/662659851 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fd874052340 con 0x7fd88013f710 2024-03-25T19:45:30.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.880+0000 7fd8677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:30.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.880+0000 7fd8677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:30.880 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:45:30.880 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:30.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 10 librados: watch_flush enter 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 10 librados: watch_flush exit 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 10 client.4443.objecter successfully canceled tick 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 >> v1:172.21.15.71:6800/3911735624 conn(0x7fd86c06ddd0 legacy=0x7fd86c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 10 monclient: shutdown 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 >> v1:172.21.15.71:6789/0 conn(0x7fd88013f710 legacy=0x7fd8801d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 shutdown_connections 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 >> 172.21.15.71:0/662659851 conn(0x7fd8800b7ea0 msgr2=0x7fd880074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 shutdown_connections 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 1 -- 172.21.15.71:0/662659851 wait complete. 2024-03-25T19:45:30.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:30.882+0000 7fd885455640 1 librados: shutdown 2024-03-25T19:45:30.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:37.700 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.799+0000 7fb9da3c9640 10 monclient: get_monmap_and_config 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.799+0000 7fb9da3c9640 10 monclient: build_initial_monmap 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.799+0000 7fb9da3c9640 10 monclient: monmap: 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:45:37.800344+0000 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:45:37.800344+0000 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:37.799 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:37.800 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:37.800 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:45:37.800 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:45:37.800 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.800+0000 7fb9da3c9640 1 Processor -- start 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.800+0000 7fb9da3c9640 1 -- start start 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.800+0000 7fb9da3c9640 10 monclient: init 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 10 monclient(hunting): picked mon.noname-a con 0x7fb9d40b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 10 monclient(hunting): picked mon.noname-c con 0x7fb9d40b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 10 monclient(hunting): picked mon.noname-b con 0x7fb9d414be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40af940 con 0x7fb9d40b4d60 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40b0a70 con 0x7fb9d40b1730 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d40b0d50 con 0x7fb9d414be90 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:37.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.801+0000 7fb9da3c9640 10 monclient(hunting): authenticate will time out at 5144.113167s 2024-03-25T19:45:37.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.805+0000 7fb9d37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fb9d40b1730 0x7fb9d40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:33638/0 (socket says 172.21.15.71:33638) 2024-03-25T19:45:37.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.805+0000 7fb9d37fe640 1 -- 172.21.15.71:0/1254065336 learned_addr learned my addr 172.21.15.71:0/1254065336 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:37.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.832+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664556299 0 0) 0x7fb9d40b0d50 con 0x7fb9d414be90 2024-03-25T19:45:37.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 10 monclient(hunting): my global_id is 14447 2024-03-25T19:45:37.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9bc003520 con 0x7fb9d414be90 2024-03-25T19:45:37.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676832232 0 0) 0x7fb9d40b0a70 con 0x7fb9d40b1730 2024-03-25T19:45:37.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:45:37.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.833+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d40b0d50 con 0x7fb9d40b1730 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3437358199 0 0) 0x7fb9bc003520 con 0x7fb9d414be90 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 >> v1:172.21.15.193:6789/0 conn(0x7fb9d40b1730 legacy=0x7fb9d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 >> v1:172.21.15.71:6789/0 conn(0x7fb9d40b4d60 legacy=0x7fb9d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d40b1030 con 0x7fb9d414be90 2024-03-25T19:45:37.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb9cc001a10 con 0x7fb9d414be90 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9d2ffd640 10 monclient: dump: 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:37.840 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:37.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9da3c9640 5 monclient: authenticate success, global_id 14447 2024-03-25T19:45:37.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.839+0000 7fb9da3c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:37.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9cc001b70 con 0x7fb9d414be90 2024-03-25T19:45:37.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:37.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9d2ffd640 1 -- 172.21.15.71:0/1254065336 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb9cc001f80 con 0x7fb9d414be90 2024-03-25T19:45:37.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 4 set_mon_vals no callback set 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9d2ffd640 10 monclient: dump: 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:37.853 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 10 monclient: shutdown 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1254065336 >> v1:172.21.15.71:6790/0 conn(0x7fb9d414be90 legacy=0x7fb9d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1254065336 shutdown_connections 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1254065336 >> 172.21.15.71:0/1254065336 conn(0x7fb9d40617a0 msgr2=0x7fb9d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1254065336 shutdown_connections 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.853+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1254065336 wait complete. 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.854+0000 7fb9da3c9640 10 monclient: build_initial_monmap 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.854+0000 7fb9da3c9640 10 monclient: monmap: 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:37.856 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.854+0000 7fb9da3c9640 1 librados: starting msgr at 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.854+0000 7fb9da3c9640 1 librados: starting objecter 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 1 Processor -- start 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 1 -- start start 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 1 librados: setting wanted keys 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 1 librados: calling monclient init 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 10 monclient: init 2024-03-25T19:45:37.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 10 monclient(hunting): picked mon.noname-c con 0x7fb9d40b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 10 monclient(hunting): picked mon.noname-b con 0x7fb9d40b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 10 monclient(hunting): picked mon.noname-a con 0x7fb9d414be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d4137ff0 con 0x7fb9d414be90 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.855+0000 7fb9da3c9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d41366d0 con 0x7fb9d40b4d60 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.856+0000 7fb9da3c9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d4137c80 con 0x7fb9d40b1730 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.856+0000 7fb9da3c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:37.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.856+0000 7fb9da3c9640 10 monclient(hunting): authenticate will time out at 5144.167894s 2024-03-25T19:45:37.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.863+0000 7fb9d3fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb9d40b1730 0x7fb9d41d7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:44502/0 (socket says 172.21.15.71:44502) 2024-03-25T19:45:37.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.863+0000 7fb9d3fff640 1 -- 172.21.15.71:0/1612098704 learned_addr learned my addr 172.21.15.71:0/1612098704 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:37.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.868+0000 7fb9d0ff9640 10 client.?.objecter ms_handle_connect 0x7fb9d40b1730 2024-03-25T19:45:37.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.868+0000 7fb9d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:37.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.875+0000 7fb9d0ff9640 10 client.?.objecter ms_handle_connect 0x7fb9d40b4d60 2024-03-25T19:45:37.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.875+0000 7fb9d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:37.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.880+0000 7fb9d0ff9640 10 client.?.objecter ms_handle_connect 0x7fb9d414be90 2024-03-25T19:45:37.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.880+0000 7fb9d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:37.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.880+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302475044 0 0) 0x7fb9d4137c80 con 0x7fb9d40b1730 2024-03-25T19:45:37.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.880+0000 7fb9d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.880+0000 7fb9d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.880+0000 7fb9d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.880+0000 7fb9d0ff9640 10 monclient(hunting): my global_id is 14450 2024-03-25T19:45:37.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.880+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c0004eb0 con 0x7fb9d40b1730 2024-03-25T19:45:37.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.892+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495521486 0 0) 0x7fb9d4137ff0 con 0x7fb9d414be90 2024-03-25T19:45:37.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.892+0000 7fb9d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.892+0000 7fb9d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.892+0000 7fb9d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.892+0000 7fb9d0ff9640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:45:37.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.892+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d4137c80 con 0x7fb9d414be90 2024-03-25T19:45:37.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.894+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 841560697 0 0) 0x7fb9c0004eb0 con 0x7fb9d40b1730 2024-03-25T19:45:37.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.894+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9d4137ff0 con 0x7fb9d40b1730 2024-03-25T19:45:37.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.894+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb9c40019d0 con 0x7fb9d40b1730 2024-03-25T19:45:37.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.894+0000 7fb9d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.894+0000 7fb9d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.894+0000 7fb9d0ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:37.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.894+0000 7fb9d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:37.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.897+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552542756 0 0) 0x7fb9d41366d0 con 0x7fb9d40b4d60 2024-03-25T19:45:37.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.897+0000 7fb9d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.897+0000 7fb9d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.897+0000 7fb9d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.897+0000 7fb9d0ff9640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:45:37.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.897+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c0004eb0 con 0x7fb9d40b4d60 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.902+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2369400891 0 0) 0x7fb9d4137c80 con 0x7fb9d414be90 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.902+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9d41366d0 con 0x7fb9d414be90 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.902+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb9cc001e80 con 0x7fb9d414be90 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.902+0000 7fb9d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.902+0000 7fb9d0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.902+0000 7fb9d0ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:37.902 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.903 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:37.903 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:37.903 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:37.903 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:37.903 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2478010592 0 0) 0x7fb9d4137ff0 con 0x7fb9d40b1730 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 >> v1:172.21.15.193:6789/0 conn(0x7fb9d40b4d60 legacy=0x7fb9d4142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 >> v1:172.21.15.71:6789/0 conn(0x7fb9d414be90 legacy=0x7fb9d4135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 1 monclient: found mon.c 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d4138480 con 0x7fb9d40b1730 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:37.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 5 monclient: authenticate success, global_id 14450 2024-03-25T19:45:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 10 monclient: _renew_subs 2024-03-25T19:45:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:45:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d41386d0 con 0x7fb9d40b1730 2024-03-25T19:45:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 10 client.14450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 10 monclient: _renew_subs 2024-03-25T19:45:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:45:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d4141570 con 0x7fb9d40b1730 2024-03-25T19:45:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.909+0000 7fb9da3c9640 1 librados: init done 2024-03-25T19:45:37.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.910+0000 7fb9da3c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:37.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.910+0000 7fb9da3c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:37.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.910+0000 7fb9da3c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:45:37.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:37.910+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d4143250 con 0x7fb9d40b1730 2024-03-25T19:45:38.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.217+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9c4003610 con 0x7fb9d40b1730 2024-03-25T19:45:38.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.217+0000 7fb9d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:38.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.217+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb9c4003800 con 0x7fb9d40b1730 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.217+0000 7fb9d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.217+0000 7fb9d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.217+0000 7fb9d0ff9640 10 monclient: dump: 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:38.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.217+0000 7fb9d27fc640 4 set_mon_vals no callback set 2024-03-25T19:45:38.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.218+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fb9c4003a20 con 0x7fb9d40b1730 2024-03-25T19:45:38.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.219+0000 7fb9d37fe640 10 monclient: get_auth_request con 0x7fb9c006dfb0 auth_method 0 2024-03-25T19:45:38.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.219+0000 7fb9d0ff9640 10 client.14450.objecter ms_handle_connect 0x7fb9c006dfb0 2024-03-25T19:45:38.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.227+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fb9c4054620 con 0x7fb9d40b1730 2024-03-25T19:45:38.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.227+0000 7fb9d0ff9640 10 client.14450.objecter ms_dispatch 0x7fb9d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:38.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.227+0000 7fb9d0ff9640 3 client.14450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:38.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.227+0000 7fb9d0ff9640 3 client.14450.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:38.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.227+0000 7fb9d0ff9640 20 client.14450.objecter dump_active .. 0 homeless 2024-03-25T19:45:38.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.227+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9c4050130 con 0x7fb9d40b1730 2024-03-25T19:45:38.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.227+0000 7fb9d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:38.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.227+0000 7fb9d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:38.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.476+0000 7fb9da3c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:38.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.476+0000 7fb9da3c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:38.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.476+0000 7fb9da3c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:45:38.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.476+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9d41375f0 con 0x7fb9d40b1730 2024-03-25T19:45:38.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.517+0000 7fb9d0ff9640 1 -- 172.21.15.71:0/1612098704 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fb9c4052060 con 0x7fb9d40b1730 2024-03-25T19:45:38.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.517+0000 7fb9d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:38.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.517+0000 7fb9d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:38.517 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:45:38.517 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:38.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 10 librados: watch_flush enter 2024-03-25T19:45:38.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 10 librados: watch_flush exit 2024-03-25T19:45:38.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 20 client.14450.objecter shutdown clearing up homeless session... 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 10 client.14450.objecter successfully canceled tick 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 >> v1:172.21.15.71:6800/3911735624 conn(0x7fb9c006dfb0 legacy=0x7fb9c0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 10 monclient: shutdown 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 >> v1:172.21.15.71:6790/0 conn(0x7fb9d40b1730 legacy=0x7fb9d41d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 shutdown_connections 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 >> 172.21.15.71:0/1612098704 conn(0x7fb9d40617a0 msgr2=0x7fb9d413c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 shutdown_connections 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.520+0000 7fb9da3c9640 1 -- 172.21.15.71:0/1612098704 wait complete. 2024-03-25T19:45:38.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:38.521+0000 7fb9da3c9640 1 librados: shutdown 2024-03-25T19:45:38.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:42.086 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:24.842 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.134 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:24.890 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.181 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:24.900 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.181 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:24.937 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.193 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:24.912 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.199 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:24.955 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.215 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:24.971 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.229 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:24.949 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:42.245 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:25.001 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.340 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.455+0000 7fc08362a640 10 monclient: get_monmap_and_config 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.455+0000 7fc08362a640 10 monclient: build_initial_monmap 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.455+0000 7fc08362a640 10 monclient: monmap: 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:45:45.457439+0000 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:45:45.457439+0000 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:45:45.457 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:45.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.456+0000 7fc08362a640 1 Processor -- start 2024-03-25T19:45:45.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.456+0000 7fc08362a640 1 -- start start 2024-03-25T19:45:45.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.456+0000 7fc08362a640 10 monclient: init 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 10 monclient(hunting): picked mon.noname-b con 0x7fc07c0a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 10 monclient(hunting): picked mon.noname-a con 0x7fc07c13f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 10 monclient(hunting): picked mon.noname-c con 0x7fc07c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c142520 con 0x7fc07c13f710 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c0b28a0 con 0x7fc07c1430c0 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c0b2a30 con 0x7fc07c0a98f0 2024-03-25T19:45:45.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:45.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.457+0000 7fc08362a640 10 monclient(hunting): authenticate will time out at 5151.770468s 2024-03-25T19:45:45.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.473+0000 7fc081bc7640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc07c1430c0 0x7fc07c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:38148/0 (socket says 172.21.15.71:38148) 2024-03-25T19:45:45.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.473+0000 7fc081bc7640 1 -- 172.21.15.71:0/3928826174 learned_addr learned my addr 172.21.15.71:0/3928826174 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:45.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.484+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275007828 0 0) 0x7fc07c0b2a30 con 0x7fc07c0a98f0 2024-03-25T19:45:45.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.484+0000 7fc073fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.484+0000 7fc073fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.484+0000 7fc073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.484+0000 7fc073fff640 10 monclient(hunting): my global_id is 14453 2024-03-25T19:45:45.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.484+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc068003520 con 0x7fc07c0a98f0 2024-03-25T19:45:45.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.496+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331982565 0 0) 0x7fc07c0b28a0 con 0x7fc07c1430c0 2024-03-25T19:45:45.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.496+0000 7fc073fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.496+0000 7fc073fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.496+0000 7fc073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.496+0000 7fc073fff640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:45:45.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.496+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc07c0b2a30 con 0x7fc07c1430c0 2024-03-25T19:45:45.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301222056 0 0) 0x7fc068003520 con 0x7fc07c0a98f0 2024-03-25T19:45:45.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 >> v1:172.21.15.193:6789/0 conn(0x7fc07c1430c0 legacy=0x7fc07c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 >> v1:172.21.15.71:6789/0 conn(0x7fc07c13f710 legacy=0x7fc07c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 1 monclient: found mon.noname-b 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c0b44c0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc078001690 con 0x7fc07c0a98f0 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc073fff640 10 monclient: dump: 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:45.501 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc08362a640 5 monclient: authenticate success, global_id 14453 2024-03-25T19:45:45.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.499+0000 7fc08362a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:45.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.518+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0780017f0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.518+0000 7fc073fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:45.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.518+0000 7fc073fff640 1 -- 172.21.15.71:0/3928826174 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc078001c00 con 0x7fc07c0a98f0 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.518+0000 7fc08362a640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.518+0000 7fc08362a640 4 set_mon_vals no callback set 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.518+0000 7fc073fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.518+0000 7fc073fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.518+0000 7fc073fff640 10 monclient: dump: 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:45.519 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.519+0000 7fc08362a640 10 monclient: shutdown 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.519+0000 7fc08362a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.519+0000 7fc08362a640 1 -- 172.21.15.71:0/3928826174 >> v1:172.21.15.71:6790/0 conn(0x7fc07c0a98f0 legacy=0x7fc07c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.519+0000 7fc08362a640 1 -- 172.21.15.71:0/3928826174 shutdown_connections 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.519+0000 7fc08362a640 1 -- 172.21.15.71:0/3928826174 >> 172.21.15.71:0/3928826174 conn(0x7fc07c0b7ea0 msgr2=0x7fc07c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.519+0000 7fc08362a640 1 -- 172.21.15.71:0/3928826174 shutdown_connections 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.519+0000 7fc08362a640 1 -- 172.21.15.71:0/3928826174 wait complete. 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 10 monclient: build_initial_monmap 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 10 monclient: monmap: 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:45.521 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 1 librados: starting msgr at 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 1 librados: starting objecter 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 1 Processor -- start 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 1 -- start start 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 1 librados: setting wanted keys 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 1 librados: calling monclient init 2024-03-25T19:45:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.520+0000 7fc08362a640 10 monclient: init 2024-03-25T19:45:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 10 monclient(hunting): picked mon.noname-a con 0x7fc07c0a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 10 monclient(hunting): picked mon.noname-b con 0x7fc07c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 10 monclient(hunting): picked mon.noname-c con 0x7fc07c1430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c1db7d0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c0b1420 con 0x7fc07c13f710 2024-03-25T19:45:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c0b15b0 con 0x7fc07c1430c0 2024-03-25T19:45:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.521+0000 7fc08362a640 10 monclient(hunting): authenticate will time out at 5151.834308s 2024-03-25T19:45:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.531+0000 7fc0813c6640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc07c0a98f0 0x7fc07c1daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48648/0 (socket says 172.21.15.71:48648) 2024-03-25T19:45:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.531+0000 7fc0813c6640 1 -- 172.21.15.71:0/1572955676 learned_addr learned my addr 172.21.15.71:0/1572955676 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.533+0000 7fc071ffb640 10 client.?.objecter ms_handle_connect 0x7fc07c0a98f0 2024-03-25T19:45:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.533+0000 7fc071ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:45.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.536+0000 7fc071ffb640 10 client.?.objecter ms_handle_connect 0x7fc07c1430c0 2024-03-25T19:45:45.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.536+0000 7fc071ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:45.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.538+0000 7fc071ffb640 10 client.?.objecter ms_handle_connect 0x7fc07c13f710 2024-03-25T19:45:45.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.538+0000 7fc071ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440417030 0 0) 0x7fc07c0b15b0 con 0x7fc07c1430c0 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 10 monclient(hunting): my global_id is 14456 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc060004eb0 con 0x7fc07c1430c0 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822113010 0 0) 0x7fc07c1db7d0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:45:45.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.540+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc07c0b15b0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.552+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164662679 0 0) 0x7fc07c0b1420 con 0x7fc07c13f710 2024-03-25T19:45:45.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.552+0000 7fc071ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.552+0000 7fc071ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.552+0000 7fc071ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.552+0000 7fc071ffb640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:45:45.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.552+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc07c1db7d0 con 0x7fc07c13f710 2024-03-25T19:45:45.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.561+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3663203620 0 0) 0x7fc060004eb0 con 0x7fc07c1430c0 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.561+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc07c0b1420 con 0x7fc07c1430c0 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.561+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc074001b00 con 0x7fc07c1430c0 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.561+0000 7fc071ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.561+0000 7fc071ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.561+0000 7fc071ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:45.562 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:45.563 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:45.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.561+0000 7fc071ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:45.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.568+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 539717296 0 0) 0x7fc07c0b15b0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.568+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc060004eb0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.568+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc078001690 con 0x7fc07c0a98f0 2024-03-25T19:45:45.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.568+0000 7fc071ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.568+0000 7fc071ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.568+0000 7fc071ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:45.569 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:45.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.570+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 510071843 0 0) 0x7fc07c1db7d0 con 0x7fc07c13f710 2024-03-25T19:45:45.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.570+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc07c0b15b0 con 0x7fc07c13f710 2024-03-25T19:45:45.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.570+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc06c001860 con 0x7fc07c13f710 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.570+0000 7fc071ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.570+0000 7fc071ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.570+0000 7fc071ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:45.571 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.589+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3363019368 0 0) 0x7fc060004eb0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.589+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 >> v1:172.21.15.71:6790/0 conn(0x7fc07c1430c0 legacy=0x7fc07c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.589+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 >> v1:172.21.15.193:6789/0 conn(0x7fc07c13f710 legacy=0x7fc07c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.589+0000 7fc071ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.589+0000 7fc071ffb640 1 monclient: found mon.a 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.589+0000 7fc071ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc071ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c0b1740 con 0x7fc07c0a98f0 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc071ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc071ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc071ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:45:45.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 10 monclient: _renew_subs 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07c1db1a0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 10 monclient: _renew_subs 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07c0b08f0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.590+0000 7fc08362a640 1 librados: init done 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.591+0000 7fc08362a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.591+0000 7fc08362a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:45.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.591+0000 7fc08362a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:45.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.591+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07c0627b0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.895+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0780038a0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.895+0000 7fc071ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:45.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.895+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc078003bf0 con 0x7fc07c0a98f0 2024-03-25T19:45:45.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.895+0000 7fc071ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.895+0000 7fc071ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:45.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.895+0000 7fc071ffb640 10 monclient: dump: 2024-03-25T19:45:45.896 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:45.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.895+0000 7fc0737fe640 4 set_mon_vals no callback set 2024-03-25T19:45:45.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.896+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fc078003e10 con 0x7fc07c0a98f0 2024-03-25T19:45:45.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.896+0000 7fc080bc5640 10 monclient: get_auth_request con 0x7fc06006e160 auth_method 0 2024-03-25T19:45:45.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.896+0000 7fc071ffb640 10 client.4455.objecter ms_handle_connect 0x7fc06006e160 2024-03-25T19:45:45.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.902+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fc078054880 con 0x7fc07c0a98f0 2024-03-25T19:45:45.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.902+0000 7fc071ffb640 10 client.4455.objecter ms_dispatch 0x7fc07c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:45.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.902+0000 7fc071ffb640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:45.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.902+0000 7fc071ffb640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:45.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.902+0000 7fc071ffb640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T19:45:45.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.902+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc078050390 con 0x7fc07c0a98f0 2024-03-25T19:45:45.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.902+0000 7fc071ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:45.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:45.902+0000 7fc071ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:46.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.127+0000 7fc08362a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:46.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.127+0000 7fc08362a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:46.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.127+0000 7fc08362a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:45:46.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.127+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc07c0abae0 con 0x7fc07c0a98f0 2024-03-25T19:45:46.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.150+0000 7fc071ffb640 1 -- 172.21.15.71:0/1572955676 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fc0780522c0 con 0x7fc07c0a98f0 2024-03-25T19:45:46.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.150+0000 7fc071ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:46.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.150+0000 7fc071ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:46.151 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:45:46.151 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.152+0000 7fc08362a640 10 librados: watch_flush enter 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.152+0000 7fc08362a640 10 librados: watch_flush exit 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.152+0000 7fc08362a640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.152+0000 7fc08362a640 10 client.4455.objecter successfully canceled tick 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 >> v1:172.21.15.71:6800/3911735624 conn(0x7fc06006e160 legacy=0x7fc060070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 10 monclient: shutdown 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 >> v1:172.21.15.71:6789/0 conn(0x7fc07c0a98f0 legacy=0x7fc07c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 shutdown_connections 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 >> 172.21.15.71:0/1572955676 conn(0x7fc07c0b7ea0 msgr2=0x7fc07c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:46.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 shutdown_connections 2024-03-25T19:45:46.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 1 -- 172.21.15.71:0/1572955676 wait complete. 2024-03-25T19:45:46.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:46.153+0000 7fc08362a640 1 librados: shutdown 2024-03-25T19:45:46.167 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:52.974 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.093+0000 7feb76972640 10 monclient: get_monmap_and_config 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.093+0000 7feb76972640 10 monclient: build_initial_monmap 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.093+0000 7feb76972640 10 monclient: monmap: 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:45:53.094343+0000 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:45:53.094343+0000 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:53.093 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:53.094 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:53.094 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:45:53.094 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:45:53.094 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:53.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.094+0000 7feb76972640 1 Processor -- start 2024-03-25T19:45:53.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.094+0000 7feb76972640 1 -- start start 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.094+0000 7feb76972640 10 monclient: init 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 10 monclient(hunting): picked mon.noname-b con 0x7feb700a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 10 monclient(hunting): picked mon.noname-c con 0x7feb7013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 10 monclient(hunting): picked mon.noname-a con 0x7feb701430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb70142520 con 0x7feb701430c0 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb700b28a0 con 0x7feb7013f710 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb700b2a30 con 0x7feb700a98f0 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:53.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.095+0000 7feb76972640 10 monclient(hunting): authenticate will time out at 5159.407541s 2024-03-25T19:45:53.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.102+0000 7feb74f0f640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7feb701430c0 0x7feb7014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52642/0 (socket says 172.21.15.71:52642) 2024-03-25T19:45:53.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.102+0000 7feb74f0f640 1 -- 172.21.15.71:0/919610704 learned_addr learned my addr 172.21.15.71:0/919610704 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:53.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.113+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365418486 0 0) 0x7feb700b2a30 con 0x7feb700a98f0 2024-03-25T19:45:53.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.113+0000 7feb6effd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:53.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.113+0000 7feb6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:53.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.113+0000 7feb6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:53.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.113+0000 7feb6effd640 10 monclient(hunting): my global_id is 14459 2024-03-25T19:45:53.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.113+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb58003520 con 0x7feb700a98f0 2024-03-25T19:45:53.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.115+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340773301 0 0) 0x7feb70142520 con 0x7feb701430c0 2024-03-25T19:45:53.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.115+0000 7feb6effd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:53.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.115+0000 7feb6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:53.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.115+0000 7feb6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:53.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.115+0000 7feb6effd640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:45:53.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.115+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb700b2a30 con 0x7feb701430c0 2024-03-25T19:45:53.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2504494371 0 0) 0x7feb58003520 con 0x7feb700a98f0 2024-03-25T19:45:53.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 >> v1:172.21.15.193:6789/0 conn(0x7feb7013f710 legacy=0x7feb701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 >> v1:172.21.15.71:6789/0 conn(0x7feb701430c0 legacy=0x7feb7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 1 monclient: found mon.noname-b 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb700b44c0 con 0x7feb700a98f0 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 10 monclient: _finish_auth 0 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7feb60001690 con 0x7feb700a98f0 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb6effd640 10 monclient: dump: 2024-03-25T19:45:53.121 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb76972640 5 monclient: authenticate success, global_id 14459 2024-03-25T19:45:53.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.120+0000 7feb76972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:53.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.135+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb600017f0 con 0x7feb700a98f0 2024-03-25T19:45:53.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.135+0000 7feb6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:53.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.135+0000 7feb6effd640 1 -- 172.21.15.71:0/919610704 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7feb60001c00 con 0x7feb700a98f0 2024-03-25T19:45:53.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.135+0000 7feb6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:53.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.135+0000 7feb6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.135+0000 7feb6effd640 10 monclient: dump: 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:53.136 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.135+0000 7feb76972640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.135+0000 7feb76972640 4 set_mon_vals no callback set 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.136+0000 7feb76972640 10 monclient: shutdown 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.136+0000 7feb76972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.136+0000 7feb76972640 1 -- 172.21.15.71:0/919610704 >> v1:172.21.15.71:6790/0 conn(0x7feb700a98f0 legacy=0x7feb700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.136+0000 7feb76972640 1 -- 172.21.15.71:0/919610704 shutdown_connections 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.136+0000 7feb76972640 1 -- 172.21.15.71:0/919610704 >> 172.21.15.71:0/919610704 conn(0x7feb700b7ea0 msgr2=0x7feb700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.136+0000 7feb76972640 1 -- 172.21.15.71:0/919610704 shutdown_connections 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.136+0000 7feb76972640 1 -- 172.21.15.71:0/919610704 wait complete. 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 10 monclient: build_initial_monmap 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 10 monclient: monmap: 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:45:53.139 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 1 librados: starting msgr at 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 1 librados: starting objecter 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 1 Processor -- start 2024-03-25T19:45:53.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 1 -- start start 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 1 librados: setting wanted keys 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 1 librados: calling monclient init 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.137+0000 7feb76972640 10 monclient: init 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 10 monclient(hunting): picked mon.noname-b con 0x7feb700a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 10 monclient(hunting): picked mon.noname-a con 0x7feb7013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 10 monclient(hunting): picked mon.noname-c con 0x7feb701430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb701370a0 con 0x7feb7013f710 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb70135090 con 0x7feb700a98f0 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb70135220 con 0x7feb701430c0 2024-03-25T19:45:53.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.138+0000 7feb76972640 10 monclient(hunting): authenticate will time out at 5159.450399s 2024-03-25T19:45:53.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.152+0000 7feb74f0f640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7feb701430c0 0x7feb701d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34042/0 (socket says 172.21.15.71:34042) 2024-03-25T19:45:53.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.152+0000 7feb74f0f640 1 -- 172.21.15.71:0/1672268469 learned_addr learned my addr 172.21.15.71:0/1672268469 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:45:53.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.155+0000 7feb6cff9640 10 client.?.objecter ms_handle_connect 0x7feb700a98f0 2024-03-25T19:45:53.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.155+0000 7feb6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:53.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.158+0000 7feb6cff9640 10 client.?.objecter ms_handle_connect 0x7feb7013f710 2024-03-25T19:45:53.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.158+0000 7feb6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:53.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.159+0000 7feb6cff9640 10 client.?.objecter ms_handle_connect 0x7feb701430c0 2024-03-25T19:45:53.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.159+0000 7feb6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:53.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.166+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895071412 0 0) 0x7feb70135220 con 0x7feb701430c0 2024-03-25T19:45:53.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.166+0000 7feb6cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:53.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.166+0000 7feb6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:53.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.166+0000 7feb6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:53.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.166+0000 7feb6cff9640 10 monclient(hunting): my global_id is 14462 2024-03-25T19:45:53.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.166+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb5c004eb0 con 0x7feb701430c0 2024-03-25T19:45:53.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.172+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557753578 0 0) 0x7feb70135090 con 0x7feb700a98f0 2024-03-25T19:45:53.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.172+0000 7feb6cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:53.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.172+0000 7feb6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:53.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.172+0000 7feb6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:53.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.172+0000 7feb6cff9640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:45:53.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.172+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70135220 con 0x7feb700a98f0 2024-03-25T19:45:53.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.176+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172910582 0 0) 0x7feb701370a0 con 0x7feb7013f710 2024-03-25T19:45:53.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.176+0000 7feb6cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:53.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.176+0000 7feb6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:53.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.176+0000 7feb6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:53.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.176+0000 7feb6cff9640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:45:53.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.176+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70135090 con 0x7feb7013f710 2024-03-25T19:45:53.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.204+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2524574558 0 0) 0x7feb70135220 con 0x7feb700a98f0 2024-03-25T19:45:53.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.204+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb701370a0 con 0x7feb700a98f0 2024-03-25T19:45:53.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.204+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7feb60001b00 con 0x7feb700a98f0 2024-03-25T19:45:53.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.204+0000 7feb6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:53.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:53.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 10 monclient(hunting): dump: 2024-03-25T19:45:53.205 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:53.205 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:53.205 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.206 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.206 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:53.206 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:53.206 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:53.206 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:53.206 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:53.206 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:53.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:53.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2761441759 0 0) 0x7feb5c004eb0 con 0x7feb701430c0 2024-03-25T19:45:53.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb70135220 con 0x7feb701430c0 2024-03-25T19:45:53.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7feb68001c10 con 0x7feb701430c0 2024-03-25T19:45:53.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:53.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:53.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.205+0000 7feb6cff9640 10 monclient(hunting): dump: 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:53.208 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:53.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.209+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1342699417 0 0) 0x7feb70135090 con 0x7feb7013f710 2024-03-25T19:45:53.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.209+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb5c004eb0 con 0x7feb7013f710 2024-03-25T19:45:53.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.209+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7feb64001860 con 0x7feb7013f710 2024-03-25T19:45:53.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.209+0000 7feb6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:53.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.209+0000 7feb6cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:53.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.209+0000 7feb6cff9640 10 monclient(hunting): dump: 2024-03-25T19:45:53.209 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:53.209 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:53.210 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.210 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.210 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:53.210 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:53.210 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:53.210 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:53.210 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:53.210 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:53.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1165619266 0 0) 0x7feb701370a0 con 0x7feb700a98f0 2024-03-25T19:45:53.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 >> v1:172.21.15.71:6790/0 conn(0x7feb701430c0 legacy=0x7feb701d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 >> v1:172.21.15.71:6789/0 conn(0x7feb7013f710 legacy=0x7feb701d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:53.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 1 monclient: found mon.b 2024-03-25T19:45:53.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb701d5000 con 0x7feb700a98f0 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 10 monclient: _renew_subs 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb701d52b0 con 0x7feb700a98f0 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 10 monclient: _renew_subs 2024-03-25T19:45:53.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:53.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb70135910 con 0x7feb700a98f0 2024-03-25T19:45:53.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.222+0000 7feb76972640 1 librados: init done 2024-03-25T19:45:53.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.223+0000 7feb76972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:53.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.223+0000 7feb76972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:53.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.223+0000 7feb76972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:53.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.223+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb700627b0 con 0x7feb700a98f0 2024-03-25T19:45:53.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.526+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb600038a0 con 0x7feb700a98f0 2024-03-25T19:45:53.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.526+0000 7feb6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:53.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.526+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7feb60001250 con 0x7feb700a98f0 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.526+0000 7feb6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.526+0000 7feb6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.526+0000 7feb6cff9640 10 monclient: dump: 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:45:53.527 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:45:53.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.526+0000 7feb6e7fc640 4 set_mon_vals no callback set 2024-03-25T19:45:53.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.527+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7feb60019a20 con 0x7feb700a98f0 2024-03-25T19:45:53.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.527+0000 7feb6f7fe640 10 monclient: get_auth_request con 0x7feb5c06e0f0 auth_method 0 2024-03-25T19:45:53.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.528+0000 7feb6cff9640 10 client.4459.objecter ms_handle_connect 0x7feb5c06e0f0 2024-03-25T19:45:53.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.533+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7feb60053640 con 0x7feb700a98f0 2024-03-25T19:45:53.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.533+0000 7feb6cff9640 10 client.4459.objecter ms_dispatch 0x7feb70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:53.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.533+0000 7feb6cff9640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:53.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.533+0000 7feb6cff9640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:53.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.534+0000 7feb6cff9640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T19:45:53.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.535+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb6004ef80 con 0x7feb700a98f0 2024-03-25T19:45:53.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.535+0000 7feb6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:53.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.535+0000 7feb6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:53.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.797+0000 7feb76972640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:53.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.797+0000 7feb76972640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:53.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.797+0000 7feb76972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:45:53.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.797+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb7013bfe0 con 0x7feb700a98f0 2024-03-25T19:45:53.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.829+0000 7feb6cff9640 1 -- 172.21.15.71:0/1672268469 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7feb60051080 con 0x7feb700a98f0 2024-03-25T19:45:53.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.829+0000 7feb6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:53.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.829+0000 7feb6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:53.829 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:45:53.830 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.831+0000 7feb76972640 10 librados: watch_flush enter 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.831+0000 7feb76972640 10 librados: watch_flush exit 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.831+0000 7feb76972640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.831+0000 7feb76972640 10 client.4459.objecter successfully canceled tick 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 >> v1:172.21.15.71:6800/3911735624 conn(0x7feb5c06e0f0 legacy=0x7feb5c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 10 monclient: shutdown 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 >> v1:172.21.15.193:6789/0 conn(0x7feb700a98f0 legacy=0x7feb701dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 shutdown_connections 2024-03-25T19:45:53.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 >> 172.21.15.71:0/1672268469 conn(0x7feb700b7ea0 msgr2=0x7feb70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:53.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 shutdown_connections 2024-03-25T19:45:53.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 1 -- 172.21.15.71:0/1672268469 wait complete. 2024-03-25T19:45:53.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:45:53.832+0000 7feb76972640 1 librados: shutdown 2024-03-25T19:45:53.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:00.650 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:00.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.752+0000 7fb0a8d4b640 10 monclient: get_monmap_and_config 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.752+0000 7fb0a8d4b640 10 monclient: build_initial_monmap 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.752+0000 7fb0a8d4b640 10 monclient: monmap: 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:46:00.752933+0000 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:46:00.752933+0000 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:46:00.752 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.752+0000 7fb0a8d4b640 1 Processor -- start 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.752+0000 7fb0a8d4b640 1 -- start start 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.752+0000 7fb0a8d4b640 10 monclient: init 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0a40b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0a40b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0a414be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a40af940 con 0x7fb0a40b1730 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a40b0a70 con 0x7fb0a40b4d60 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a40b0d50 con 0x7fb0a414be90 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:00.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a8d4b640 10 monclient(hunting): authenticate will time out at 5167.065229s 2024-03-25T19:46:00.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a2d9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb0a414be90 0x7fb0a414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33396/0 (socket says 172.21.15.71:33396) 2024-03-25T19:46:00.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.753+0000 7fb0a2d9d640 1 -- 172.21.15.71:0/2428168989 learned_addr learned my addr 172.21.15.71:0/2428168989 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:00.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.788+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433599031 0 0) 0x7fb0a40af940 con 0x7fb0a40b1730 2024-03-25T19:46:00.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.788+0000 7fb0a159a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.788+0000 7fb0a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.788+0000 7fb0a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.788+0000 7fb0a159a640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:46:00.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.788+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb08c003520 con 0x7fb0a40b1730 2024-03-25T19:46:00.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62060502 0 0) 0x7fb08c003520 con 0x7fb0a40b1730 2024-03-25T19:46:00.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 >> v1:172.21.15.71:6790/0 conn(0x7fb0a414be90 legacy=0x7fb0a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 >> v1:172.21.15.193:6789/0 conn(0x7fb0a40b4d60 legacy=0x7fb0a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 10 monclient: _finish_hunting 0 2024-03-25T19:46:00.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 1 monclient: found mon.noname-a 2024-03-25T19:46:00.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:00.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a40b1030 con 0x7fb0a40b1730 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 10 monclient: _finish_auth 0 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 10 monclient: _check_auth_tickets 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb098001690 con 0x7fb0a40b1730 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a159a640 10 monclient: dump: 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.801 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:00.802 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:00.802 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:00.802 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:00.802 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:00.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a8d4b640 5 monclient: authenticate success, global_id 4464 2024-03-25T19:46:00.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.800+0000 7fb0a8d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:00.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0980017f0 con 0x7fb0a40b1730 2024-03-25T19:46:00.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a159a640 1 -- 172.21.15.71:0/2428168989 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb098001c00 con 0x7fb0a40b1730 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a159a640 10 monclient: dump: 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:00.823 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:00.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a8d4b640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:00.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a8d4b640 4 set_mon_vals no callback set 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a8d4b640 10 monclient: shutdown 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a8d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.823+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2428168989 >> v1:172.21.15.71:6789/0 conn(0x7fb0a40b1730 legacy=0x7fb0a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.824+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2428168989 shutdown_connections 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.824+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2428168989 >> 172.21.15.71:0/2428168989 conn(0x7fb0a40617a0 msgr2=0x7fb0a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.824+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2428168989 shutdown_connections 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.824+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2428168989 wait complete. 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 10 monclient: build_initial_monmap 2024-03-25T19:46:00.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 10 monclient: monmap: 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 1 librados: starting msgr at 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 1 librados: starting objecter 2024-03-25T19:46:00.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 1 Processor -- start 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 1 -- start start 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 1 librados: setting wanted keys 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 1 librados: calling monclient init 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.825+0000 7fb0a8d4b640 10 monclient: init 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0a40b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb0a40b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0a414be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a4140380 con 0x7fb0a414be90 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a4141ca0 con 0x7fb0a40b4d60 2024-03-25T19:46:00.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a4140860 con 0x7fb0a40b1730 2024-03-25T19:46:00.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:00.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.826+0000 7fb0a8d4b640 10 monclient(hunting): authenticate will time out at 5167.137946s 2024-03-25T19:46:00.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.837+0000 7fb0a2d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb0a414be90 0x7fb0a41e22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33986/0 (socket says 172.21.15.71:33986) 2024-03-25T19:46:00.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.837+0000 7fb0a2d9d640 1 -- 172.21.15.71:0/2073770314 learned_addr learned my addr 172.21.15.71:0/2073770314 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:00.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.839+0000 7fb092ffd640 10 client.?.objecter ms_handle_connect 0x7fb0a414be90 2024-03-25T19:46:00.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.839+0000 7fb092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:00.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.844+0000 7fb092ffd640 10 client.?.objecter ms_handle_connect 0x7fb0a40b4d60 2024-03-25T19:46:00.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.844+0000 7fb092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:00.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.849+0000 7fb092ffd640 10 client.?.objecter ms_handle_connect 0x7fb0a40b1730 2024-03-25T19:46:00.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.849+0000 7fb092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:00.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.858+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436108890 0 0) 0x7fb0a4140380 con 0x7fb0a414be90 2024-03-25T19:46:00.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.858+0000 7fb092ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.858+0000 7fb092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.858+0000 7fb092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.858+0000 7fb092ffd640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:46:00.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.858+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb084004eb0 con 0x7fb0a414be90 2024-03-25T19:46:00.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.866+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212356636 0 0) 0x7fb0a4141ca0 con 0x7fb0a40b4d60 2024-03-25T19:46:00.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.866+0000 7fb092ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.866+0000 7fb092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.866+0000 7fb092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.866+0000 7fb092ffd640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:46:00.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.866+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a4140380 con 0x7fb0a40b4d60 2024-03-25T19:46:00.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.872+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964085240 0 0) 0x7fb0a4140860 con 0x7fb0a40b1730 2024-03-25T19:46:00.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.872+0000 7fb092ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.872+0000 7fb092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.872+0000 7fb092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.872+0000 7fb092ffd640 10 monclient(hunting): my global_id is 14468 2024-03-25T19:46:00.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.872+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a4141ca0 con 0x7fb0a40b1730 2024-03-25T19:46:00.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 723884637 0 0) 0x7fb0a4140380 con 0x7fb0a40b4d60 2024-03-25T19:46:00.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0a4140860 con 0x7fb0a40b4d60 2024-03-25T19:46:00.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb098001b80 con 0x7fb0a40b4d60 2024-03-25T19:46:00.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 10 monclient(hunting): dump: 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:00.882 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:00.883 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:00.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:00.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3422641391 0 0) 0x7fb084004eb0 con 0x7fb0a414be90 2024-03-25T19:46:00.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0a4140380 con 0x7fb0a414be90 2024-03-25T19:46:00.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb09c001b10 con 0x7fb0a414be90 2024-03-25T19:46:00.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:00.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.881+0000 7fb092ffd640 10 monclient(hunting): dump: 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:00.884 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:00.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1850392698 0 0) 0x7fb0a4141ca0 con 0x7fb0a40b1730 2024-03-25T19:46:00.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb084004eb0 con 0x7fb0a40b1730 2024-03-25T19:46:00.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb094001860 con 0x7fb0a40b1730 2024-03-25T19:46:00.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:00.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 10 monclient(hunting): dump: 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2081641493 0 0) 0x7fb0a4140860 con 0x7fb0a40b4d60 2024-03-25T19:46:00.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 >> v1:172.21.15.71:6790/0 conn(0x7fb0a40b1730 legacy=0x7fb0a41db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 >> v1:172.21.15.71:6789/0 conn(0x7fb0a414be90 legacy=0x7fb0a41e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 1 monclient: found mon.b 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a41423a0 con 0x7fb0a40b4d60 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:00.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb0a8d4b640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb0a8d4b640 10 monclient: _renew_subs 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb0a8d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.896+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a4138650 con 0x7fb0a40b4d60 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.897+0000 7fb0a8d4b640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.897+0000 7fb0a8d4b640 10 monclient: _renew_subs 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.897+0000 7fb0a8d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.897+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a4140d00 con 0x7fb0a40b4d60 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.897+0000 7fb0a8d4b640 1 librados: init done 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.898+0000 7fb0a8d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:00.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.898+0000 7fb0a8d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:00.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.898+0000 7fb0a8d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:00.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:00.898+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a40627b0 con 0x7fb0a40b4d60 2024-03-25T19:46:01.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.206+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0980038a0 con 0x7fb0a40b4d60 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.207+0000 7fb092ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.207+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb098003bb0 con 0x7fb0a40b4d60 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.207+0000 7fb092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.207+0000 7fb092ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.207+0000 7fb092ffd640 10 monclient: dump: 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:01.207 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:01.208 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:01.208 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:01.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.207+0000 7fb0a0d99640 4 set_mon_vals no callback set 2024-03-25T19:46:01.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.207+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fb098019b80 con 0x7fb0a40b4d60 2024-03-25T19:46:01.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.208+0000 7fb0a259c640 10 monclient: get_auth_request con 0x7fb08406e0f0 auth_method 0 2024-03-25T19:46:01.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.208+0000 7fb092ffd640 10 client.4465.objecter ms_handle_connect 0x7fb08406e0f0 2024-03-25T19:46:01.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.213+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fb098053700 con 0x7fb0a40b4d60 2024-03-25T19:46:01.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.213+0000 7fb092ffd640 10 client.4465.objecter ms_dispatch 0x7fb0a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:01.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.213+0000 7fb092ffd640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:01.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.213+0000 7fb092ffd640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:01.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.213+0000 7fb092ffd640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T19:46:01.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.215+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb09804f210 con 0x7fb0a40b4d60 2024-03-25T19:46:01.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.215+0000 7fb092ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:01.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.215+0000 7fb092ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:01.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.466+0000 7fb0a8d4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:01.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.466+0000 7fb0a8d4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:01.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.466+0000 7fb0a8d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:01.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.466+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0a41387e0 con 0x7fb0a40b4d60 2024-03-25T19:46:01.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.494+0000 7fb092ffd640 1 -- 172.21.15.71:0/2073770314 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fb098051140 con 0x7fb0a40b4d60 2024-03-25T19:46:01.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.494+0000 7fb092ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:01.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.494+0000 7fb092ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:01.494 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:46:01.494 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:01.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 10 librados: watch_flush enter 2024-03-25T19:46:01.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 10 librados: watch_flush exit 2024-03-25T19:46:01.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 10 client.4465.objecter successfully canceled tick 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 >> v1:172.21.15.71:6800/3911735624 conn(0x7fb08406e0f0 legacy=0x7fb0840705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 10 monclient: shutdown 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 >> v1:172.21.15.193:6789/0 conn(0x7fb0a40b4d60 legacy=0x7fb0a41db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 shutdown_connections 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 >> 172.21.15.71:0/2073770314 conn(0x7fb0a40617a0 msgr2=0x7fb0a413c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 shutdown_connections 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 1 -- 172.21.15.71:0/2073770314 wait complete. 2024-03-25T19:46:01.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:01.496+0000 7fb0a8d4b640 1 librados: shutdown 2024-03-25T19:46:01.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:02.157 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:44.913 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.208 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:44.964 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.267 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:45.023 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.270 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:05:45.027 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.271 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:44.990 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.283 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:45.002 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.283 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:45.039 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.316 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:45.035 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.316 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:05:45.073 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:08.315 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:08.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.429+0000 7f192fef1640 10 monclient: get_monmap_and_config 2024-03-25T19:46:08.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.429+0000 7f192fef1640 10 monclient: build_initial_monmap 2024-03-25T19:46:08.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.429+0000 7f192fef1640 10 monclient: monmap: 2024-03-25T19:46:08.429 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:46:08.430486+0000 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:46:08.430486+0000 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:46:08.430 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:08.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.430+0000 7f192fef1640 1 Processor -- start 2024-03-25T19:46:08.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.430+0000 7f192fef1640 1 -- start start 2024-03-25T19:46:08.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.430+0000 7f192fef1640 10 monclient: init 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 10 monclient(hunting): picked mon.noname-c con 0x7f19280a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 10 monclient(hunting): picked mon.noname-b con 0x7f192813f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 10 monclient(hunting): picked mon.noname-a con 0x7f19281430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1928142520 con 0x7f19281430c0 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280b28a0 con 0x7f19280a98f0 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280b2a30 con 0x7f192813f710 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:08.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.431+0000 7f192fef1640 10 monclient(hunting): authenticate will time out at 5174.743311s 2024-03-25T19:46:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.442+0000 7f192d48c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f192813f710 0x7f19281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50344/0 (socket says 172.21.15.71:50344) 2024-03-25T19:46:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.442+0000 7f192d48c640 1 -- 172.21.15.71:0/2515001844 learned_addr learned my addr 172.21.15.71:0/2515001844 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.452+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708765498 0 0) 0x7f19280b2a30 con 0x7f192813f710 2024-03-25T19:46:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.452+0000 7f192cc8b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.452+0000 7f192cc8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.452+0000 7f192cc8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.452+0000 7f192cc8b640 10 monclient(hunting): my global_id is 14471 2024-03-25T19:46:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.452+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1914003520 con 0x7f192813f710 2024-03-25T19:46:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.456+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298223017 0 0) 0x7f19280b28a0 con 0x7f19280a98f0 2024-03-25T19:46:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.456+0000 7f192cc8b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.456+0000 7f192cc8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.456+0000 7f192cc8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.456+0000 7f192cc8b640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:46:08.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.456+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19280b2a30 con 0x7f19280a98f0 2024-03-25T19:46:08.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.458+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538918883 0 0) 0x7f1928142520 con 0x7f19281430c0 2024-03-25T19:46:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.458+0000 7f192cc8b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.458+0000 7f192cc8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.458+0000 7f192cc8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.458+0000 7f192cc8b640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:46:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.458+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19280b28a0 con 0x7f19281430c0 2024-03-25T19:46:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242362733 0 0) 0x7f19280b2a30 con 0x7f19280a98f0 2024-03-25T19:46:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 >> v1:172.21.15.71:6790/0 conn(0x7f192813f710 legacy=0x7f19281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 >> v1:172.21.15.71:6789/0 conn(0x7f19281430c0 legacy=0x7f192814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 10 monclient: _finish_hunting 0 2024-03-25T19:46:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 1 monclient: found mon.noname-c 2024-03-25T19:46:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280b44c0 con 0x7f19280a98f0 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 10 monclient: _finish_auth 0 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 10 monclient: _check_auth_tickets 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1924001670 con 0x7f19280a98f0 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192cc8b640 10 monclient: dump: 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:08.469 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:08.470 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:08.470 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:08.470 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:08.470 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:08.470 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:08.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192fef1640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:46:08.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.468+0000 7f192fef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:08.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.471+0000 7f192cc8b640 1 -- 172.21.15.71:0/2515001844 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19240017d0 con 0x7f19280a98f0 2024-03-25T19:46:08.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.471+0000 7f192cc8b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:08.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.471+0000 7f192fef1640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:08.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.471+0000 7f192fef1640 4 set_mon_vals no callback set 2024-03-25T19:46:08.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.472+0000 7f192fef1640 10 monclient: shutdown 2024-03-25T19:46:08.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.472+0000 7f192fef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:08.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.472+0000 7f192fef1640 1 -- 172.21.15.71:0/2515001844 >> v1:172.21.15.193:6789/0 conn(0x7f19280a98f0 legacy=0x7f19280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.472+0000 7f192fef1640 1 -- 172.21.15.71:0/2515001844 shutdown_connections 2024-03-25T19:46:08.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.472+0000 7f192fef1640 1 -- 172.21.15.71:0/2515001844 >> 172.21.15.71:0/2515001844 conn(0x7f19280b7ea0 msgr2=0x7f19280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:08.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.473+0000 7f192fef1640 1 -- 172.21.15.71:0/2515001844 shutdown_connections 2024-03-25T19:46:08.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.473+0000 7f192fef1640 1 -- 172.21.15.71:0/2515001844 wait complete. 2024-03-25T19:46:08.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.473+0000 7f192fef1640 10 monclient: build_initial_monmap 2024-03-25T19:46:08.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.474+0000 7f192fef1640 10 monclient: monmap: 2024-03-25T19:46:08.473 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:08.473 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:08.473 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:46:08.473 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:46:08.473 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:08.474 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:46:08.474 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:08.474 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:46:08.474 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:46:08.474 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:08.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.474+0000 7f192fef1640 1 librados: starting msgr at 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.474+0000 7f192fef1640 1 librados: starting objecter 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.474+0000 7f192fef1640 1 Processor -- start 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.474+0000 7f192fef1640 1 -- start start 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.474+0000 7f192fef1640 1 librados: setting wanted keys 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.474+0000 7f192fef1640 1 librados: calling monclient init 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.474+0000 7f192fef1640 10 monclient: init 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 10 monclient(hunting): picked mon.noname-a con 0x7f19280a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 10 monclient(hunting): picked mon.noname-c con 0x7f192813f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 10 monclient(hunting): picked mon.noname-b con 0x7f19281430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f192813b0a0 con 0x7f19280a98f0 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f192813b310 con 0x7f19281430c0 2024-03-25T19:46:08.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f192813b5f0 con 0x7f192813f710 2024-03-25T19:46:08.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:08.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.475+0000 7f192fef1640 10 monclient(hunting): authenticate will time out at 5174.787648s 2024-03-25T19:46:08.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.487+0000 7f192e48e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f19281430c0 0x7f19281e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:42648/0 (socket says 172.21.15.71:42648) 2024-03-25T19:46:08.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.487+0000 7f192e48e640 1 -- 172.21.15.71:0/789168983 learned_addr learned my addr 172.21.15.71:0/789168983 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:08.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.490+0000 7f191e7fc640 10 client.?.objecter ms_handle_connect 0x7f19281430c0 2024-03-25T19:46:08.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.490+0000 7f191e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:08.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.496+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161471359 0 0) 0x7f192813b310 con 0x7f19281430c0 2024-03-25T19:46:08.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.496+0000 7f191e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.496+0000 7f191e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.496+0000 7f191e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.496+0000 7f191e7fc640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:46:08.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.496+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f190c004eb0 con 0x7f19281430c0 2024-03-25T19:46:08.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.497+0000 7f191e7fc640 10 client.?.objecter ms_handle_connect 0x7f19280a98f0 2024-03-25T19:46:08.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.497+0000 7f191e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:08.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.497+0000 7f191e7fc640 10 client.?.objecter ms_handle_connect 0x7f192813f710 2024-03-25T19:46:08.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.497+0000 7f191e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1309519119 0 0) 0x7f190c004eb0 con 0x7f19281430c0 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f192813b310 con 0x7f19281430c0 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1920001c70 con 0x7f19281430c0 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 10 monclient(hunting): dump: 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:08.521 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602155495 0 0) 0x7f192813b0a0 con 0x7f19280a98f0 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:46:08.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.521+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f190c004eb0 con 0x7f19280a98f0 2024-03-25T19:46:08.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.522+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964623402 0 0) 0x7f192813b5f0 con 0x7f192813f710 2024-03-25T19:46:08.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.522+0000 7f191e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.522+0000 7f191e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.522+0000 7f191e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.522+0000 7f191e7fc640 10 monclient(hunting): my global_id is 14474 2024-03-25T19:46:08.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.522+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f192813b0a0 con 0x7f192813f710 2024-03-25T19:46:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2273216194 0 0) 0x7f192813b310 con 0x7f19281430c0 2024-03-25T19:46:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 >> v1:172.21.15.71:6790/0 conn(0x7f192813f710 legacy=0x7f19281df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 >> v1:172.21.15.71:6789/0 conn(0x7f19280a98f0 legacy=0x7f192813ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 1 monclient: found mon.b 2024-03-25T19:46:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280b70e0 con 0x7f19281430c0 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f191e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 5 monclient: authenticate success, global_id 4471 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 10 monclient: _renew_subs 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19280b78a0 con 0x7f19281430c0 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 10 monclient: _renew_subs 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f192812f630 con 0x7f19281430c0 2024-03-25T19:46:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.535+0000 7f192fef1640 1 librados: init done 2024-03-25T19:46:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.536+0000 7f192fef1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.536+0000 7f192fef1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.536+0000 7f192fef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.536+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192814f2d0 con 0x7f19281430c0 2024-03-25T19:46:08.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.807+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1920003970 con 0x7f19281430c0 2024-03-25T19:46:08.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.807+0000 7f191e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:08.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.807+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1920003dc0 con 0x7f19281430c0 2024-03-25T19:46:08.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.807+0000 7f191e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:08.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.807+0000 7f191e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:08.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.807+0000 7f191e7fc640 10 monclient: dump: 2024-03-25T19:46:08.807 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:08.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.807+0000 7f191ffff640 4 set_mon_vals no callback set 2024-03-25T19:46:08.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.808+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f1920003fe0 con 0x7f19281430c0 2024-03-25T19:46:08.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.808+0000 7f192dc8d640 10 monclient: get_auth_request con 0x7f190c06de80 auth_method 0 2024-03-25T19:46:08.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.809+0000 7f191e7fc640 10 client.4471.objecter ms_handle_connect 0x7f190c06de80 2024-03-25T19:46:08.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.814+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f1920054b00 con 0x7f19281430c0 2024-03-25T19:46:08.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.814+0000 7f191e7fc640 10 client.4471.objecter ms_dispatch 0x7f1928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:08.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.814+0000 7f191e7fc640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:08.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.814+0000 7f191e7fc640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:08.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.814+0000 7f191e7fc640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-25T19:46:08.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.814+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1920050610 con 0x7f19281430c0 2024-03-25T19:46:08.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.814+0000 7f191e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:08.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:08.814+0000 7f191e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:09.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.038+0000 7f192fef1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:09.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.039+0000 7f192fef1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:09.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.039+0000 7f192fef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:09.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.039+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19280627b0 con 0x7f19281430c0 2024-03-25T19:46:09.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.063+0000 7f191e7fc640 1 -- 172.21.15.71:0/789168983 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f1920052540 con 0x7f19281430c0 2024-03-25T19:46:09.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.063+0000 7f191e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:09.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.063+0000 7f191e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:09.064 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:46:09.064 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:09.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 10 librados: watch_flush enter 2024-03-25T19:46:09.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 10 librados: watch_flush exit 2024-03-25T19:46:09.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-25T19:46:09.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 10 client.4471.objecter successfully canceled tick 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 >> v1:172.21.15.71:6800/3911735624 conn(0x7f190c06de80 legacy=0x7f190c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 10 monclient: shutdown 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 >> v1:172.21.15.193:6789/0 conn(0x7f19281430c0 legacy=0x7f19281e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 shutdown_connections 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 >> 172.21.15.71:0/789168983 conn(0x7f19280b7ea0 msgr2=0x7f1928074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 shutdown_connections 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 1 -- 172.21.15.71:0/789168983 wait complete. 2024-03-25T19:46:09.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:09.066+0000 7f192fef1640 1 librados: shutdown 2024-03-25T19:46:09.078 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:15.886 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:15.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:15.999+0000 7fb6ffff4640 10 monclient: get_monmap_and_config 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:15.999+0000 7fb6ffff4640 10 monclient: build_initial_monmap 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:15.999+0000 7fb6ffff4640 10 monclient: monmap: 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:46:16.000004+0000 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:46:16.000004+0000 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:15.999 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:46:16.000 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:46:16.000 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:15.999+0000 7fb6ffff4640 1 Processor -- start 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 1 -- start start 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 10 monclient: init 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f80a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f813f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f81430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f8142520 con 0x7fb6f80a98f0 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f80b28a0 con 0x7fb6f81430c0 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f80b2a30 con 0x7fb6f813f710 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:16.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.000+0000 7fb6ffff4640 10 monclient(hunting): authenticate will time out at 5182.312634s 2024-03-25T19:46:16.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.012+0000 7fb6fe591640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fb6f81430c0 0x7fb6f814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:47414/0 (socket says 172.21.15.71:47414) 2024-03-25T19:46:16.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.012+0000 7fb6fe591640 1 -- 172.21.15.71:0/2065768767 learned_addr learned my addr 172.21.15.71:0/2065768767 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:16.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.034+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238098168 0 0) 0x7fb6f80b28a0 con 0x7fb6f81430c0 2024-03-25T19:46:16.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.034+0000 7fb6fcd8e640 10 monclient(hunting): _negotiate 2024-03-25T19:46:16.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.034+0000 7fb6fcd8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:16.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.034+0000 7fb6fcd8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:16.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.034+0000 7fb6fcd8e640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:46:16.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.034+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e4003520 con 0x7fb6f81430c0 2024-03-25T19:46:16.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.041+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088837671 0 0) 0x7fb6f80b2a30 con 0x7fb6f813f710 2024-03-25T19:46:16.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.041+0000 7fb6fcd8e640 10 monclient(hunting): _negotiate 2024-03-25T19:46:16.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.041+0000 7fb6fcd8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:16.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.041+0000 7fb6fcd8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:16.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.041+0000 7fb6fcd8e640 10 monclient(hunting): my global_id is 14477 2024-03-25T19:46:16.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.041+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f80b28a0 con 0x7fb6f813f710 2024-03-25T19:46:16.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.044+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111922746 0 0) 0x7fb6f8142520 con 0x7fb6f80a98f0 2024-03-25T19:46:16.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.044+0000 7fb6fcd8e640 10 monclient(hunting): _negotiate 2024-03-25T19:46:16.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.044+0000 7fb6fcd8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:16.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.044+0000 7fb6fcd8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:16.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.044+0000 7fb6fcd8e640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:46:16.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.044+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f80b2a30 con 0x7fb6f80a98f0 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497431917 0 0) 0x7fb6e4003520 con 0x7fb6f81430c0 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 >> v1:172.21.15.71:6790/0 conn(0x7fb6f813f710 legacy=0x7fb6f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 >> v1:172.21.15.71:6789/0 conn(0x7fb6f80a98f0 legacy=0x7fb6f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 10 monclient: _finish_hunting 0 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 1 monclient: found mon.noname-c 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f80b44c0 con 0x7fb6f81430c0 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 10 monclient: _finish_auth 0 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 10 monclient: _check_auth_tickets 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb6f0001a10 con 0x7fb6f81430c0 2024-03-25T19:46:16.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6ffff4640 5 monclient: authenticate success, global_id 4474 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6ffff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6fcd8e640 10 monclient: dump: 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:16.048 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:16.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.047+0000 7fb6ffff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.081+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6f0001b70 con 0x7fb6f81430c0 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.081+0000 7fb6fcd8e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.081+0000 7fb6fcd8e640 1 -- 172.21.15.71:0/2065768767 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb6f0001f80 con 0x7fb6f81430c0 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.081+0000 7fb6ffff4640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.081+0000 7fb6ffff4640 4 set_mon_vals no callback set 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.081+0000 7fb6fcd8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.081+0000 7fb6fcd8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.081+0000 7fb6fcd8e640 10 monclient: dump: 2024-03-25T19:46:16.081 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:16.082 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.082+0000 7fb6ffff4640 10 monclient: shutdown 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.082+0000 7fb6ffff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.082+0000 7fb6ffff4640 1 -- 172.21.15.71:0/2065768767 >> v1:172.21.15.193:6789/0 conn(0x7fb6f81430c0 legacy=0x7fb6f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.082+0000 7fb6ffff4640 1 -- 172.21.15.71:0/2065768767 shutdown_connections 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.082+0000 7fb6ffff4640 1 -- 172.21.15.71:0/2065768767 >> 172.21.15.71:0/2065768767 conn(0x7fb6f80b7ea0 msgr2=0x7fb6f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.082+0000 7fb6ffff4640 1 -- 172.21.15.71:0/2065768767 shutdown_connections 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.082+0000 7fb6ffff4640 1 -- 172.21.15.71:0/2065768767 wait complete. 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 10 monclient: build_initial_monmap 2024-03-25T19:46:16.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 10 monclient: monmap: 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 1 librados: starting msgr at 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 1 librados: starting objecter 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 1 Processor -- start 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 1 -- start start 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 1 librados: setting wanted keys 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 1 librados: calling monclient init 2024-03-25T19:46:16.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.083+0000 7fb6ffff4640 10 monclient: init 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f80a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f81430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f8070450 con 0x7fb6f813f710 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f81ded90 con 0x7fb6f81430c0 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f806e820 con 0x7fb6f80a98f0 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:16.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.084+0000 7fb6ffff4640 10 monclient(hunting): authenticate will time out at 5182.396516s 2024-03-25T19:46:16.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.093+0000 7fb6fdd90640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb6f80a98f0 0x7fb6f806e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55176/0 (socket says 172.21.15.71:55176) 2024-03-25T19:46:16.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.093+0000 7fb6fdd90640 1 -- 172.21.15.71:0/934996540 learned_addr learned my addr 172.21.15.71:0/934996540 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:16.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.095+0000 7fb6ee7fc640 10 client.?.objecter ms_handle_connect 0x7fb6f80a98f0 2024-03-25T19:46:16.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.095+0000 7fb6ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:16.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.096+0000 7fb6ee7fc640 10 client.?.objecter ms_handle_connect 0x7fb6f813f710 2024-03-25T19:46:16.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.096+0000 7fb6ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:16.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.098+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171647688 0 0) 0x7fb6f806e820 con 0x7fb6f80a98f0 2024-03-25T19:46:16.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.098+0000 7fb6ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:16.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.098+0000 7fb6ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:16.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.098+0000 7fb6ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:16.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.098+0000 7fb6ee7fc640 10 monclient(hunting): my global_id is 14480 2024-03-25T19:46:16.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.098+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc004eb0 con 0x7fb6f80a98f0 2024-03-25T19:46:16.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.103+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908103021 0 0) 0x7fb6f8070450 con 0x7fb6f813f710 2024-03-25T19:46:16.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.103+0000 7fb6ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:16.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.103+0000 7fb6ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:16.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.103+0000 7fb6ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:16.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.103+0000 7fb6ee7fc640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:46:16.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.103+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f806e820 con 0x7fb6f813f710 2024-03-25T19:46:16.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.105+0000 7fb6ee7fc640 10 client.?.objecter ms_handle_connect 0x7fb6f81430c0 2024-03-25T19:46:16.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.105+0000 7fb6ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:16.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.107+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2567210920 0 0) 0x7fb6dc004eb0 con 0x7fb6f80a98f0 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.107+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6f8070450 con 0x7fb6f80a98f0 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.107+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb6f4001a10 con 0x7fb6f80a98f0 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.107+0000 7fb6ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.107+0000 7fb6ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.107+0000 7fb6ee7fc640 10 monclient(hunting): dump: 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:16.107 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:16.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.107+0000 7fb6ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:16.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3746460761 0 0) 0x7fb6f8070450 con 0x7fb6f80a98f0 2024-03-25T19:46:16.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 >> v1:172.21.15.193:6789/0 conn(0x7fb6f81430c0 legacy=0x7fb6f81e6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 >> v1:172.21.15.71:6789/0 conn(0x7fb6f813f710 legacy=0x7fb6f81df290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 1 monclient: found mon.c 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f8139d20 con 0x7fb6f80a98f0 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 5 monclient: authenticate success, global_id 14480 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 10 monclient: _renew_subs 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f81df9a0 con 0x7fb6f80a98f0 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 10 client.14480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:16.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 10 monclient: _renew_subs 2024-03-25T19:46:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f806f230 con 0x7fb6f80a98f0 2024-03-25T19:46:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.113+0000 7fb6ffff4640 1 librados: init done 2024-03-25T19:46:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.114+0000 7fb6ffff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.114+0000 7fb6ffff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.114+0000 7fb6ffff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.114+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f8139f10 con 0x7fb6f80a98f0 2024-03-25T19:46:16.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.412+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6f4001c70 con 0x7fb6f80a98f0 2024-03-25T19:46:16.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.412+0000 7fb6ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.412+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb6f4003a60 con 0x7fb6f80a98f0 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.412+0000 7fb6ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.412+0000 7fb6ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.412+0000 7fb6ee7fc640 10 monclient: dump: 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:16.413 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:16.414 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:16.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.412+0000 7fb6effff640 4 set_mon_vals no callback set 2024-03-25T19:46:16.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.413+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fb6f4003c80 con 0x7fb6f80a98f0 2024-03-25T19:46:16.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.414+0000 7fb6fd58f640 10 monclient: get_auth_request con 0x7fb6dc06dda0 auth_method 0 2024-03-25T19:46:16.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.414+0000 7fb6ee7fc640 10 client.14480.objecter ms_handle_connect 0x7fb6dc06dda0 2024-03-25T19:46:16.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.419+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fb6f40548b0 con 0x7fb6f80a98f0 2024-03-25T19:46:16.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.419+0000 7fb6ee7fc640 10 client.14480.objecter ms_dispatch 0x7fb6f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:16.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.419+0000 7fb6ee7fc640 3 client.14480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:16.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.419+0000 7fb6ee7fc640 3 client.14480.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:16.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.419+0000 7fb6ee7fc640 20 client.14480.objecter dump_active .. 0 homeless 2024-03-25T19:46:16.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.419+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6f40503c0 con 0x7fb6f80a98f0 2024-03-25T19:46:16.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.419+0000 7fb6ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:16.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.419+0000 7fb6ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:16.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.669+0000 7fb6ffff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:16.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.669+0000 7fb6ffff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:16.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.669+0000 7fb6ffff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:16.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.669+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6f81def20 con 0x7fb6f80a98f0 2024-03-25T19:46:16.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.695+0000 7fb6ee7fc640 1 -- 172.21.15.71:0/934996540 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fb6f40522f0 con 0x7fb6f80a98f0 2024-03-25T19:46:16.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.695+0000 7fb6ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:16.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.695+0000 7fb6ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:16.695 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:46:16.696 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 10 librados: watch_flush enter 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 10 librados: watch_flush exit 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 20 client.14480.objecter shutdown clearing up homeless session... 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 10 client.14480.objecter successfully canceled tick 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 >> v1:172.21.15.71:6800/3911735624 conn(0x7fb6dc06dda0 legacy=0x7fb6dc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 10 monclient: shutdown 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 >> v1:172.21.15.71:6790/0 conn(0x7fb6f80a98f0 legacy=0x7fb6f806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 shutdown_connections 2024-03-25T19:46:16.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 >> 172.21.15.71:0/934996540 conn(0x7fb6f80b7ea0 msgr2=0x7fb6f813c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:16.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 shutdown_connections 2024-03-25T19:46:16.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.697+0000 7fb6ffff4640 1 -- 172.21.15.71:0/934996540 wait complete. 2024-03-25T19:46:16.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:16.698+0000 7fb6ffff4640 1 librados: shutdown 2024-03-25T19:46:16.708 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:22.256 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:05.012 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:22.261 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:05.017 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:22.288 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:05.045 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:22.317 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:05.037 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:22.325 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:05.044 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:22.351 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:05.108 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:22.353 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:05.073 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:22.356 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:05.112 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:22.386 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:05.142 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:23.515 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.633+0000 7fe28afc2640 10 monclient: get_monmap_and_config 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.633+0000 7fe28afc2640 10 monclient: build_initial_monmap 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.633+0000 7fe28afc2640 10 monclient: monmap: 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:46:23.634270+0000 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:46:23.634270+0000 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:46:23.633 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:46:23.634 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 1 Processor -- start 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 1 -- start start 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 10 monclient: init 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 10 monclient(hunting): picked mon.noname-c con 0x7fe2840a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 10 monclient(hunting): picked mon.noname-b con 0x7fe28413f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 10 monclient(hunting): picked mon.noname-a con 0x7fe2841430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.634+0000 7fe28afc2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe284142520 con 0x7fe2841430c0 2024-03-25T19:46:23.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.635+0000 7fe28afc2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2840b28a0 con 0x7fe2840a98f0 2024-03-25T19:46:23.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.635+0000 7fe28afc2640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2840b2a30 con 0x7fe28413f710 2024-03-25T19:46:23.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.635+0000 7fe28afc2640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:23.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.635+0000 7fe28afc2640 10 monclient(hunting): authenticate will time out at 5189.946871s 2024-03-25T19:46:23.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.641+0000 7fe28955f640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe2841430c0 0x7fe28414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:38228/0 (socket says 172.21.15.71:38228) 2024-03-25T19:46:23.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.641+0000 7fe28955f640 1 -- 172.21.15.71:0/3189130659 learned_addr learned my addr 172.21.15.71:0/3189130659 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:23.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.654+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888830636 0 0) 0x7fe284142520 con 0x7fe2841430c0 2024-03-25T19:46:23.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.654+0000 7fe27b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.654+0000 7fe27b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.654+0000 7fe27b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.654+0000 7fe27b7fe640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:46:23.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.654+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c003520 con 0x7fe2841430c0 2024-03-25T19:46:23.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.663+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826244948 0 0) 0x7fe2840b2a30 con 0x7fe28413f710 2024-03-25T19:46:23.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.663+0000 7fe27b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.663+0000 7fe27b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.663+0000 7fe27b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.663+0000 7fe27b7fe640 10 monclient(hunting): my global_id is 14483 2024-03-25T19:46:23.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.663+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe284142520 con 0x7fe28413f710 2024-03-25T19:46:23.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.666+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376125603 0 0) 0x7fe2840b28a0 con 0x7fe2840a98f0 2024-03-25T19:46:23.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.666+0000 7fe27b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.666+0000 7fe27b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.666+0000 7fe27b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.666+0000 7fe27b7fe640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:46:23.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.666+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2840b2a30 con 0x7fe2840a98f0 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1815158813 0 0) 0x7fe26c003520 con 0x7fe2841430c0 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 >> v1:172.21.15.71:6790/0 conn(0x7fe28413f710 legacy=0x7fe2841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 >> v1:172.21.15.193:6789/0 conn(0x7fe2840a98f0 legacy=0x7fe2840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2840b44c0 con 0x7fe2841430c0 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe27c001a10 con 0x7fe2841430c0 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe27b7fe640 10 monclient: dump: 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe28afc2640 5 monclient: authenticate success, global_id 4482 2024-03-25T19:46:23.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.676+0000 7fe28afc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:23.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.711+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe27c001b70 con 0x7fe2841430c0 2024-03-25T19:46:23.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.711+0000 7fe27b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:23.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.711+0000 7fe27b7fe640 1 -- 172.21.15.71:0/3189130659 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe27c001f80 con 0x7fe2841430c0 2024-03-25T19:46:23.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.711+0000 7fe28afc2640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:23.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.711+0000 7fe28afc2640 4 set_mon_vals no callback set 2024-03-25T19:46:23.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.711+0000 7fe27b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.711+0000 7fe27b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.711+0000 7fe27b7fe640 10 monclient: dump: 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:23.712 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:23.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.712+0000 7fe28afc2640 10 monclient: shutdown 2024-03-25T19:46:23.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.712+0000 7fe28afc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:23.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.712+0000 7fe28afc2640 1 -- 172.21.15.71:0/3189130659 >> v1:172.21.15.71:6789/0 conn(0x7fe2841430c0 legacy=0x7fe28414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.712+0000 7fe28afc2640 1 -- 172.21.15.71:0/3189130659 shutdown_connections 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.712+0000 7fe28afc2640 1 -- 172.21.15.71:0/3189130659 >> 172.21.15.71:0/3189130659 conn(0x7fe2840b7ea0 msgr2=0x7fe2840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.712+0000 7fe28afc2640 1 -- 172.21.15.71:0/3189130659 shutdown_connections 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.712+0000 7fe28afc2640 1 -- 172.21.15.71:0/3189130659 wait complete. 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.713+0000 7fe28afc2640 10 monclient: build_initial_monmap 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.713+0000 7fe28afc2640 10 monclient: monmap: 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:23.715 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.713+0000 7fe28afc2640 1 librados: starting msgr at 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.713+0000 7fe28afc2640 1 librados: starting objecter 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 1 Processor -- start 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 1 -- start start 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 1 librados: setting wanted keys 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 1 librados: calling monclient init 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 10 monclient: init 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:23.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 10 monclient(hunting): picked mon.noname-b con 0x7fe2840a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:23.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 10 monclient(hunting): picked mon.noname-a con 0x7fe28413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:23.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 10 monclient(hunting): picked mon.noname-c con 0x7fe2841430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:23.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2840abb00 con 0x7fe28413f710 2024-03-25T19:46:23.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2840abde0 con 0x7fe2840a98f0 2024-03-25T19:46:23.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe284138e20 con 0x7fe2841430c0 2024-03-25T19:46:23.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:23.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.714+0000 7fe28afc2640 10 monclient(hunting): authenticate will time out at 5190.026820s 2024-03-25T19:46:23.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.724+0000 7fe288d5e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fe2840a98f0 0x7fe2841385e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:36190/0 (socket says 172.21.15.71:36190) 2024-03-25T19:46:23.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.724+0000 7fe288d5e640 1 -- 172.21.15.71:0/3922414080 learned_addr learned my addr 172.21.15.71:0/3922414080 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:23.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.724+0000 7fe2797fa640 10 client.?.objecter ms_handle_connect 0x7fe2840a98f0 2024-03-25T19:46:23.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.724+0000 7fe2797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:23.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.730+0000 7fe2797fa640 10 client.?.objecter ms_handle_connect 0x7fe2841430c0 2024-03-25T19:46:23.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.730+0000 7fe2797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:23.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.731+0000 7fe2797fa640 10 client.?.objecter ms_handle_connect 0x7fe28413f710 2024-03-25T19:46:23.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.731+0000 7fe2797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:23.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.732+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603362569 0 0) 0x7fe2840abde0 con 0x7fe2840a98f0 2024-03-25T19:46:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.732+0000 7fe2797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.732+0000 7fe2797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.732+0000 7fe2797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.732+0000 7fe2797fa640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:46:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.732+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe270004eb0 con 0x7fe2840a98f0 2024-03-25T19:46:23.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.741+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 309494426 0 0) 0x7fe270004eb0 con 0x7fe2840a98f0 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.741+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2840abde0 con 0x7fe2840a98f0 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.741+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe280001a10 con 0x7fe2840a98f0 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.741+0000 7fe2797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.741+0000 7fe2797fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.741+0000 7fe2797fa640 10 monclient(hunting): dump: 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:23.742 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:23.743 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:23.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.741+0000 7fe2797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.752+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453244545 0 0) 0x7fe2840abb00 con 0x7fe28413f710 2024-03-25T19:46:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.752+0000 7fe2797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.752+0000 7fe2797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.752+0000 7fe2797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.752+0000 7fe2797fa640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:46:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.752+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe270004eb0 con 0x7fe28413f710 2024-03-25T19:46:23.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.754+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630888205 0 0) 0x7fe284138e20 con 0x7fe2841430c0 2024-03-25T19:46:23.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.754+0000 7fe2797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.754+0000 7fe2797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.754+0000 7fe2797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.754+0000 7fe2797fa640 10 monclient(hunting): my global_id is 14486 2024-03-25T19:46:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.754+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2840abb00 con 0x7fe2841430c0 2024-03-25T19:46:23.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703619389 0 0) 0x7fe2840abde0 con 0x7fe2840a98f0 2024-03-25T19:46:23.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 >> v1:172.21.15.71:6790/0 conn(0x7fe2841430c0 legacy=0x7fe2840ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 >> v1:172.21.15.71:6789/0 conn(0x7fe28413f710 legacy=0x7fe284136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:23.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 1 monclient: found mon.b 2024-03-25T19:46:23.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:23.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:23.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe284137b00 con 0x7fe2840a98f0 2024-03-25T19:46:23.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:23.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:23.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe2797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:23.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:46:23.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 10 monclient: _renew_subs 2024-03-25T19:46:23.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:23.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2841e34b0 con 0x7fe2840a98f0 2024-03-25T19:46:23.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:23.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 10 monclient: _renew_subs 2024-03-25T19:46:23.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:23.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2841e3820 con 0x7fe2840a98f0 2024-03-25T19:46:23.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.757+0000 7fe28afc2640 1 librados: init done 2024-03-25T19:46:23.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.759+0000 7fe28afc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:23.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.759+0000 7fe28afc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:23.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.759+0000 7fe28afc2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:23.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:23.759+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2840627b0 con 0x7fe2840a98f0 2024-03-25T19:46:24.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.036+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe280001cb0 con 0x7fe2840a98f0 2024-03-25T19:46:24.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.036+0000 7fe2797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:24.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.036+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe280003a60 con 0x7fe2840a98f0 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.036+0000 7fe2797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.036+0000 7fe2797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.036+0000 7fe2797fa640 10 monclient: dump: 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:24.037 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:24.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.036+0000 7fe27affd640 4 set_mon_vals no callback set 2024-03-25T19:46:24.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.037+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fe280003c80 con 0x7fe2840a98f0 2024-03-25T19:46:24.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.038+0000 7fe27bfff640 10 monclient: get_auth_request con 0x7fe27006de80 auth_method 0 2024-03-25T19:46:24.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.038+0000 7fe2797fa640 10 client.4483.objecter ms_handle_connect 0x7fe27006de80 2024-03-25T19:46:24.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.041+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fe280054830 con 0x7fe2840a98f0 2024-03-25T19:46:24.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.041+0000 7fe2797fa640 10 client.4483.objecter ms_dispatch 0x7fe284000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:24.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.041+0000 7fe2797fa640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:24.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.041+0000 7fe2797fa640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:24.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.041+0000 7fe2797fa640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T19:46:24.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.042+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe280050340 con 0x7fe2840a98f0 2024-03-25T19:46:24.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.042+0000 7fe2797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:24.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.042+0000 7fe2797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:24.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.292+0000 7fe28afc2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:24.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.292+0000 7fe28afc2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:24.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.292+0000 7fe28afc2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:24.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.292+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2840b1760 con 0x7fe2840a98f0 2024-03-25T19:46:24.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.323+0000 7fe2797fa640 1 -- 172.21.15.71:0/3922414080 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fe280052270 con 0x7fe2840a98f0 2024-03-25T19:46:24.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.323+0000 7fe2797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:24.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.323+0000 7fe2797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:24.323 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:46:24.324 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.325+0000 7fe28afc2640 10 librados: watch_flush enter 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.325+0000 7fe28afc2640 10 librados: watch_flush exit 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.325+0000 7fe28afc2640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.325+0000 7fe28afc2640 10 client.4483.objecter successfully canceled tick 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.325+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 >> v1:172.21.15.71:6800/3911735624 conn(0x7fe27006de80 legacy=0x7fe270070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.325+0000 7fe28afc2640 10 monclient: shutdown 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.325+0000 7fe28afc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.325+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 >> v1:172.21.15.193:6789/0 conn(0x7fe2840a98f0 legacy=0x7fe2841385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.326+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 shutdown_connections 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.326+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 >> 172.21.15.71:0/3922414080 conn(0x7fe2840b7ea0 msgr2=0x7fe28414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:24.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.326+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 shutdown_connections 2024-03-25T19:46:24.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.326+0000 7fe28afc2640 1 -- 172.21.15.71:0/3922414080 wait complete. 2024-03-25T19:46:24.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:24.326+0000 7fe28afc2640 1 librados: shutdown 2024-03-25T19:46:24.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:31.144 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:31.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.258+0000 7fe13a393640 10 monclient: get_monmap_and_config 2024-03-25T19:46:31.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.258+0000 7fe13a393640 10 monclient: build_initial_monmap 2024-03-25T19:46:31.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.258+0000 7fe13a393640 10 monclient: monmap: 2024-03-25T19:46:31.258 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:31.258 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:31.258 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:46:31.259460+0000 2024-03-25T19:46:31.259 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:46:31.259460+0000 2024-03-25T19:46:31.259 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:31.259 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:31.259 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:31.259 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:46:31.259 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:46:31.259 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:31.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.259+0000 7fe13a393640 1 Processor -- start 2024-03-25T19:46:31.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.259+0000 7fe13a393640 1 -- start start 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.259+0000 7fe13a393640 10 monclient: init 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 10 monclient(hunting): picked mon.noname-c con 0x7fe1340b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 10 monclient(hunting): picked mon.noname-b con 0x7fe1340b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 10 monclient(hunting): picked mon.noname-a con 0x7fe13414be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1340af940 con 0x7fe13414be90 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1340b0a70 con 0x7fe1340b4d60 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1340b0d50 con 0x7fe1340b1730 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:31.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.260+0000 7fe13a393640 10 monclient(hunting): authenticate will time out at 5197.572503s 2024-03-25T19:46:31.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.272+0000 7fe138930640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe13414be90 0x7fe13414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42682/0 (socket says 172.21.15.71:42682) 2024-03-25T19:46:31.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.272+0000 7fe138930640 1 -- 172.21.15.71:0/1501398185 learned_addr learned my addr 172.21.15.71:0/1501398185 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:31.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.279+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448316593 0 0) 0x7fe1340af940 con 0x7fe13414be90 2024-03-25T19:46:31.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.279+0000 7fe132ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:31.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.279+0000 7fe132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:31.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.279+0000 7fe132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:31.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.279+0000 7fe132ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:46:31.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.279+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe11c003520 con 0x7fe13414be90 2024-03-25T19:46:31.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.286+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130155226 0 0) 0x7fe1340b0a70 con 0x7fe1340b4d60 2024-03-25T19:46:31.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.286+0000 7fe132ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:31.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.286+0000 7fe132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:31.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.286+0000 7fe132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:31.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.286+0000 7fe132ffd640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:46:31.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.286+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1340af940 con 0x7fe1340b4d60 2024-03-25T19:46:31.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3569861838 0 0) 0x7fe11c003520 con 0x7fe13414be90 2024-03-25T19:46:31.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 >> v1:172.21.15.71:6790/0 conn(0x7fe1340b1730 legacy=0x7fe1340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 >> v1:172.21.15.193:6789/0 conn(0x7fe1340b4d60 legacy=0x7fe13413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:31.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 1 monclient: found mon.noname-a 2024-03-25T19:46:31.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:31.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1340b1030 con 0x7fe13414be90 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe12c001a10 con 0x7fe13414be90 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe132ffd640 10 monclient: dump: 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:31.291 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:31.292 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:31.292 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:31.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe13a393640 5 monclient: authenticate success, global_id 4488 2024-03-25T19:46:31.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.290+0000 7fe13a393640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:31.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe12c001b70 con 0x7fe13414be90 2024-03-25T19:46:31.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe132ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:31.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe132ffd640 1 -- 172.21.15.71:0/1501398185 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe12c001f80 con 0x7fe13414be90 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe132ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe132ffd640 10 monclient: dump: 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:31.306 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:31.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:31.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 4 set_mon_vals no callback set 2024-03-25T19:46:31.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 10 monclient: shutdown 2024-03-25T19:46:31.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:31.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 1 -- 172.21.15.71:0/1501398185 >> v1:172.21.15.71:6789/0 conn(0x7fe13414be90 legacy=0x7fe13414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 1 -- 172.21.15.71:0/1501398185 shutdown_connections 2024-03-25T19:46:31.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 1 -- 172.21.15.71:0/1501398185 >> 172.21.15.71:0/1501398185 conn(0x7fe1340617a0 msgr2=0x7fe134061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:31.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 1 -- 172.21.15.71:0/1501398185 shutdown_connections 2024-03-25T19:46:31.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.306+0000 7fe13a393640 1 -- 172.21.15.71:0/1501398185 wait complete. 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.307+0000 7fe13a393640 10 monclient: build_initial_monmap 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.307+0000 7fe13a393640 10 monclient: monmap: 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:31.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.308+0000 7fe13a393640 1 librados: starting msgr at 2024-03-25T19:46:31.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.308+0000 7fe13a393640 1 librados: starting objecter 2024-03-25T19:46:31.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.308+0000 7fe13a393640 1 Processor -- start 2024-03-25T19:46:31.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.308+0000 7fe13a393640 1 -- start start 2024-03-25T19:46:31.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.308+0000 7fe13a393640 1 librados: setting wanted keys 2024-03-25T19:46:31.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.308+0000 7fe13a393640 1 librados: calling monclient init 2024-03-25T19:46:31.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.308+0000 7fe13a393640 10 monclient: init 2024-03-25T19:46:31.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:31.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:31.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 10 monclient(hunting): picked mon.noname-b con 0x7fe1340b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:31.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 10 monclient(hunting): picked mon.noname-c con 0x7fe1340b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:31.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 10 monclient(hunting): picked mon.noname-a con 0x7fe13414be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:31.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe134141950 con 0x7fe13414be90 2024-03-25T19:46:31.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1341d7940 con 0x7fe1340b1730 2024-03-25T19:46:31.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1341d8570 con 0x7fe1340b4d60 2024-03-25T19:46:31.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:31.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.309+0000 7fe13a393640 10 monclient(hunting): authenticate will time out at 5197.621770s 2024-03-25T19:46:31.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.318+0000 7fe138930640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe13414be90 0x7fe1341e2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42696/0 (socket says 172.21.15.71:42696) 2024-03-25T19:46:31.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.318+0000 7fe138930640 1 -- 172.21.15.71:0/3836116980 learned_addr learned my addr 172.21.15.71:0/3836116980 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:31.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.319+0000 7fe130ff9640 10 client.?.objecter ms_handle_connect 0x7fe13414be90 2024-03-25T19:46:31.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.319+0000 7fe130ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:31.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.327+0000 7fe130ff9640 10 client.?.objecter ms_handle_connect 0x7fe1340b4d60 2024-03-25T19:46:31.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.327+0000 7fe130ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:31.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.329+0000 7fe130ff9640 10 client.?.objecter ms_handle_connect 0x7fe1340b1730 2024-03-25T19:46:31.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.329+0000 7fe130ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:31.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.338+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830597017 0 0) 0x7fe1341d7940 con 0x7fe1340b1730 2024-03-25T19:46:31.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.338+0000 7fe130ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:31.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.338+0000 7fe130ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:31.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.338+0000 7fe130ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:31.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.338+0000 7fe130ff9640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:46:31.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.338+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe120004eb0 con 0x7fe1340b1730 2024-03-25T19:46:31.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.339+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769042064 0 0) 0x7fe134141950 con 0x7fe13414be90 2024-03-25T19:46:31.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.339+0000 7fe130ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:31.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.339+0000 7fe130ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:31.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.339+0000 7fe130ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:31.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.339+0000 7fe130ff9640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:46:31.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.339+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1341d7940 con 0x7fe13414be90 2024-03-25T19:46:31.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.340+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949859839 0 0) 0x7fe1341d8570 con 0x7fe1340b4d60 2024-03-25T19:46:31.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.340+0000 7fe130ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:31.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.340+0000 7fe130ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:31.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.340+0000 7fe130ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:31.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.340+0000 7fe130ff9640 10 monclient(hunting): my global_id is 14492 2024-03-25T19:46:31.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.340+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe134141950 con 0x7fe1340b4d60 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.347+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120403420 0 0) 0x7fe120004eb0 con 0x7fe1340b1730 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.347+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1341d8570 con 0x7fe1340b1730 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.347+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe124001970 con 0x7fe1340b1730 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.347+0000 7fe130ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.347+0000 7fe130ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.347+0000 7fe130ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:31.347 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.348 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.348 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.348 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:31.348 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:31.348 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:31.348 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:31.348 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:31.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.347+0000 7fe130ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:31.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.354+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 948285304 0 0) 0x7fe1341d7940 con 0x7fe13414be90 2024-03-25T19:46:31.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.354+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe120004eb0 con 0x7fe13414be90 2024-03-25T19:46:31.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.354+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe12c001e80 con 0x7fe13414be90 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.354+0000 7fe130ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.354+0000 7fe130ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.354+0000 7fe130ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:31.354 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:31.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1015673279 0 0) 0x7fe134141950 con 0x7fe1340b4d60 2024-03-25T19:46:31.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1341d7940 con 0x7fe1340b4d60 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe128001860 con 0x7fe1340b4d60 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:31.358 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1030529632 0 0) 0x7fe1341d8570 con 0x7fe1340b1730 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 >> v1:172.21.15.71:6790/0 conn(0x7fe1340b4d60 legacy=0x7fe134142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 >> v1:172.21.15.71:6789/0 conn(0x7fe13414be90 legacy=0x7fe1341e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 1 monclient: found mon.b 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe134138840 con 0x7fe1340b1730 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe130ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:46:31.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 10 monclient: _renew_subs 2024-03-25T19:46:31.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:31.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe134142f70 con 0x7fe1340b1730 2024-03-25T19:46:31.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:31.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 10 monclient: _renew_subs 2024-03-25T19:46:31.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:31.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe134137da0 con 0x7fe1340b1730 2024-03-25T19:46:31.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.357+0000 7fe13a393640 1 librados: init done 2024-03-25T19:46:31.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.358+0000 7fe13a393640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:31.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.358+0000 7fe13a393640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:31.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.358+0000 7fe13a393640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:31.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.358+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1341d7ad0 con 0x7fe1340b1730 2024-03-25T19:46:31.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.631+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe124003610 con 0x7fe1340b1730 2024-03-25T19:46:31.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.631+0000 7fe130ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:31.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.631+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe124003920 con 0x7fe1340b1730 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.631+0000 7fe130ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.631+0000 7fe130ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.631+0000 7fe130ff9640 10 monclient: dump: 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:31.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.631+0000 7fe1327fc640 4 set_mon_vals no callback set 2024-03-25T19:46:31.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.632+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fe124019970 con 0x7fe1340b1730 2024-03-25T19:46:31.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.633+0000 7fe1337fe640 10 monclient: get_auth_request con 0x7fe12006e0b0 auth_method 0 2024-03-25T19:46:31.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.633+0000 7fe130ff9640 10 client.4489.objecter ms_handle_connect 0x7fe12006e0b0 2024-03-25T19:46:31.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.637+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fe1240536c0 con 0x7fe1340b1730 2024-03-25T19:46:31.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.637+0000 7fe130ff9640 10 client.4489.objecter ms_dispatch 0x7fe134000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:31.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.637+0000 7fe130ff9640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:31.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.637+0000 7fe130ff9640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:31.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.637+0000 7fe130ff9640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T19:46:31.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.637+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe12404f000 con 0x7fe1340b1730 2024-03-25T19:46:31.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.637+0000 7fe130ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:31.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.637+0000 7fe130ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:31.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.881+0000 7fe13a393640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:31.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.881+0000 7fe13a393640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:31.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.881+0000 7fe13a393640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:46:31.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.881+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1341389d0 con 0x7fe1340b1730 2024-03-25T19:46:31.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.910+0000 7fe130ff9640 1 -- 172.21.15.71:0/3836116980 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fe124051100 con 0x7fe1340b1730 2024-03-25T19:46:31.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.910+0000 7fe130ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:31.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.910+0000 7fe130ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:31.911 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:46:31.911 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:31.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 10 librados: watch_flush enter 2024-03-25T19:46:31.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 10 librados: watch_flush exit 2024-03-25T19:46:31.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T19:46:31.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 10 client.4489.objecter successfully canceled tick 2024-03-25T19:46:31.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 >> v1:172.21.15.71:6800/3911735624 conn(0x7fe12006e0b0 legacy=0x7fe120070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 10 monclient: shutdown 2024-03-25T19:46:31.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:31.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 >> v1:172.21.15.193:6789/0 conn(0x7fe1340b1730 legacy=0x7fe1341d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 shutdown_connections 2024-03-25T19:46:31.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 >> 172.21.15.71:0/3836116980 conn(0x7fe1340617a0 msgr2=0x7fe1340b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:31.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 shutdown_connections 2024-03-25T19:46:31.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 1 -- 172.21.15.71:0/3836116980 wait complete. 2024-03-25T19:46:31.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:31.913+0000 7fe13a393640 1 librados: shutdown 2024-03-25T19:46:31.925 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:38.731 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:38.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.838+0000 7fc5a5481640 10 monclient: get_monmap_and_config 2024-03-25T19:46:38.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.838+0000 7fc5a5481640 10 monclient: build_initial_monmap 2024-03-25T19:46:38.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.838+0000 7fc5a5481640 10 monclient: monmap: 2024-03-25T19:46:38.839 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:46:38.840723+0000 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:46:38.840723+0000 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:46:38.840 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:38.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.839+0000 7fc5a5481640 1 Processor -- start 2024-03-25T19:46:38.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.839+0000 7fc5a5481640 1 -- start start 2024-03-25T19:46:38.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.839+0000 7fc5a5481640 10 monclient: init 2024-03-25T19:46:38.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.839+0000 7fc5a5481640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:38.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.839+0000 7fc5a5481640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.839+0000 7fc5a5481640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a006ea40 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.839+0000 7fc5a5481640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a0071800 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.839+0000 7fc5a5481640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a0071c40 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.840+0000 7fc5a5481640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a006f1f0 con 0x7fc5a006ea40 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.840+0000 7fc5a5481640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a0070d00 con 0x7fc5a0071c40 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.840+0000 7fc5a5481640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a00720b0 con 0x7fc5a0071800 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.840+0000 7fc5a5481640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.840+0000 7fc5a5481640 10 monclient(hunting): authenticate will time out at 5205.152894s 2024-03-25T19:46:38.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.840+0000 7fc59e7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fc5a0071800 0x7fc5a0147070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42052/0 (socket says 172.21.15.71:42052) 2024-03-25T19:46:38.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.840+0000 7fc59e7fc640 1 -- 172.21.15.71:0/1055269239 learned_addr learned my addr 172.21.15.71:0/1055269239 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:38.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.891+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728387741 0 0) 0x7fc5a0070d00 con 0x7fc5a0071c40 2024-03-25T19:46:38.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.891+0000 7fc59dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:38.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.891+0000 7fc59dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:38.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.891+0000 7fc59dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:38.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.891+0000 7fc59dffb640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:46:38.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.891+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc588003520 con 0x7fc5a0071c40 2024-03-25T19:46:38.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029768226 0 0) 0x7fc588003520 con 0x7fc5a0071c40 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 >> v1:172.21.15.71:6790/0 conn(0x7fc5a0071800 legacy=0x7fc5a0147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 >> v1:172.21.15.71:6789/0 conn(0x7fc5a006ea40 legacy=0x7fc5a013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 1 monclient: found mon.noname-c 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a0072780 con 0x7fc5a0071c40 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc590001690 con 0x7fc5a0071c40 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc5a5481640 5 monclient: authenticate success, global_id 4492 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc5a5481640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.909+0000 7fc59dffb640 10 monclient: dump: 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:38.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.910+0000 7fc5a5481640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:38.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5900017f0 con 0x7fc5a0071c40 2024-03-25T19:46:38.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc59dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc59dffb640 1 -- 172.21.15.71:0/1055269239 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc590001c00 con 0x7fc5a0071c40 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc5a5481640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc5a5481640 4 set_mon_vals no callback set 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc59dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc59dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc59dffb640 10 monclient: dump: 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:38.923 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:38.925 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:38.925 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:38.925 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:38.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc5a5481640 10 monclient: shutdown 2024-03-25T19:46:38.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc5a5481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc5a5481640 1 -- 172.21.15.71:0/1055269239 >> v1:172.21.15.193:6789/0 conn(0x7fc5a0071c40 legacy=0x7fc5a014a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.922+0000 7fc5a5481640 1 -- 172.21.15.71:0/1055269239 shutdown_connections 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.923+0000 7fc5a5481640 1 -- 172.21.15.71:0/1055269239 >> 172.21.15.71:0/1055269239 conn(0x7fc5a006b630 msgr2=0x7fc5a00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.923+0000 7fc5a5481640 1 -- 172.21.15.71:0/1055269239 shutdown_connections 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.923+0000 7fc5a5481640 1 -- 172.21.15.71:0/1055269239 wait complete. 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.923+0000 7fc5a5481640 10 monclient: build_initial_monmap 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.923+0000 7fc5a5481640 10 monclient: monmap: 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:46:38.928 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.923+0000 7fc5a5481640 1 librados: starting msgr at 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.924+0000 7fc5a5481640 1 librados: starting objecter 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.924+0000 7fc5a5481640 1 Processor -- start 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.924+0000 7fc5a5481640 1 -- start start 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.924+0000 7fc5a5481640 1 librados: setting wanted keys 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.924+0000 7fc5a5481640 1 librados: calling monclient init 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.924+0000 7fc5a5481640 10 monclient: init 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.924+0000 7fc5a5481640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a006ea40 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a0071800 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:38.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a0071c40 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:38.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a011b650 con 0x7fc5a0071c40 2024-03-25T19:46:38.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a0138980 con 0x7fc5a0071800 2024-03-25T19:46:38.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a011d1c0 con 0x7fc5a006ea40 2024-03-25T19:46:38.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:38.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.925+0000 7fc5a5481640 10 monclient(hunting): authenticate will time out at 5205.238199s 2024-03-25T19:46:38.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.938+0000 7fc59f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc5a0071c40 0x7fc5a0138270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49066/0 (socket says 172.21.15.71:49066) 2024-03-25T19:46:38.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.938+0000 7fc59f7fe640 1 -- 172.21.15.71:0/2202124191 learned_addr learned my addr 172.21.15.71:0/2202124191 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:38.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.939+0000 7fc5877fe640 10 client.?.objecter ms_handle_connect 0x7fc5a0071c40 2024-03-25T19:46:38.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.939+0000 7fc5877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:38.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.941+0000 7fc5877fe640 10 client.?.objecter ms_handle_connect 0x7fc5a0071800 2024-03-25T19:46:38.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.941+0000 7fc5877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:38.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.952+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770432956 0 0) 0x7fc5a011b650 con 0x7fc5a0071c40 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.952+0000 7fc5877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.952+0000 7fc5877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.952+0000 7fc5877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.952+0000 7fc5877fe640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.952+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc58c004eb0 con 0x7fc5a0071c40 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.953+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298476018 0 0) 0x7fc5a0138980 con 0x7fc5a0071800 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.953+0000 7fc5877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.953+0000 7fc5877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.953+0000 7fc5877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.953+0000 7fc5877fe640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.953+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a011b650 con 0x7fc5a0071800 2024-03-25T19:46:38.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.954+0000 7fc5877fe640 10 client.?.objecter ms_handle_connect 0x7fc5a006ea40 2024-03-25T19:46:38.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.954+0000 7fc5877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:38.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.959+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367129729 0 0) 0x7fc5a011d1c0 con 0x7fc5a006ea40 2024-03-25T19:46:38.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.959+0000 7fc5877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:38.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.959+0000 7fc5877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:38.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.959+0000 7fc5877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:38.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.959+0000 7fc5877fe640 10 monclient(hunting): my global_id is 14498 2024-03-25T19:46:38.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.959+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a0138980 con 0x7fc5a006ea40 2024-03-25T19:46:38.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.965+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549264495 0 0) 0x7fc58c004eb0 con 0x7fc5a0071c40 2024-03-25T19:46:38.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.966+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5a011d1c0 con 0x7fc5a0071c40 2024-03-25T19:46:38.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.966+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc590001bc0 con 0x7fc5a0071c40 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.966+0000 7fc5877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.966+0000 7fc5877fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.966+0000 7fc5877fe640 10 monclient(hunting): dump: 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:38.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.966+0000 7fc5877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.971+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 990236645 0 0) 0x7fc5a0138980 con 0x7fc5a006ea40 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.971+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc58c004eb0 con 0x7fc5a006ea40 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.971+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc5980017f0 con 0x7fc5a006ea40 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.971+0000 7fc5877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.971+0000 7fc5877fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.971+0000 7fc5877fe640 10 monclient(hunting): dump: 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:38.972 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:38.973 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:38.973 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:38.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.973+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 641275201 0 0) 0x7fc5a011b650 con 0x7fc5a0071800 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.973+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5a0138980 con 0x7fc5a0071800 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.973+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc594001c20 con 0x7fc5a0071800 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.973+0000 7fc5877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.973+0000 7fc5877fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.973+0000 7fc5877fe640 10 monclient(hunting): dump: 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:38.974 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:38.975 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:38.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2935912932 0 0) 0x7fc58c004eb0 con 0x7fc5a006ea40 2024-03-25T19:46:38.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 >> v1:172.21.15.193:6789/0 conn(0x7fc5a0071800 legacy=0x7fc5a0137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 >> v1:172.21.15.71:6789/0 conn(0x7fc5a0071c40 legacy=0x7fc5a0138270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 1 monclient: found mon.c 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a011d4a0 con 0x7fc5a006ea40 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 5 monclient: authenticate success, global_id 14498 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 10 monclient: _renew_subs 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a011b8a0 con 0x7fc5a006ea40 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 10 client.14498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 10 monclient: _renew_subs 2024-03-25T19:46:38.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:38.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a01c9240 con 0x7fc5a006ea40 2024-03-25T19:46:38.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.977+0000 7fc5a5481640 1 librados: init done 2024-03-25T19:46:38.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.978+0000 7fc5a5481640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:38.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.978+0000 7fc5a5481640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:38.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.978+0000 7fc5a5481640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:38.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:38.978+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a00627b0 con 0x7fc5a006ea40 2024-03-25T19:46:39.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.276+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc598001d00 con 0x7fc5a006ea40 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.276+0000 7fc5877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.276+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc598003790 con 0x7fc5a006ea40 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.276+0000 7fc5877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.276+0000 7fc5877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.276+0000 7fc5877fe640 10 monclient: dump: 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:39.278 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:39.279 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:39.279 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:39.279 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:39.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.276+0000 7fc59d7fa640 4 set_mon_vals no callback set 2024-03-25T19:46:39.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.277+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fc5980039b0 con 0x7fc5a006ea40 2024-03-25T19:46:39.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.278+0000 7fc59e7fc640 10 monclient: get_auth_request con 0x7fc58c06e160 auth_method 0 2024-03-25T19:46:39.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.278+0000 7fc5877fe640 10 client.14498.objecter ms_handle_connect 0x7fc58c06e160 2024-03-25T19:46:39.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.285+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fc598053500 con 0x7fc5a006ea40 2024-03-25T19:46:39.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.285+0000 7fc5877fe640 10 client.14498.objecter ms_dispatch 0x7fc5a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:39.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.285+0000 7fc5877fe640 3 client.14498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:39.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.285+0000 7fc5877fe640 3 client.14498.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:39.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.285+0000 7fc5877fe640 20 client.14498.objecter dump_active .. 0 homeless 2024-03-25T19:46:39.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.285+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc59804ee40 con 0x7fc5a006ea40 2024-03-25T19:46:39.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.285+0000 7fc5877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:39.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.285+0000 7fc5877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:39.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.526+0000 7fc5a5481640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:39.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.526+0000 7fc5a5481640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:39.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.526+0000 7fc5a5481640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:39.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.526+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5a00681e0 con 0x7fc5a006ea40 2024-03-25T19:46:39.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.546+0000 7fc5877fe640 1 -- 172.21.15.71:0/2202124191 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fc598050f40 con 0x7fc5a006ea40 2024-03-25T19:46:39.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.546+0000 7fc5877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:39.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.546+0000 7fc5877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:39.548 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:46:39.548 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:39.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 10 librados: watch_flush enter 2024-03-25T19:46:39.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 10 librados: watch_flush exit 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 20 client.14498.objecter shutdown clearing up homeless session... 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 10 client.14498.objecter successfully canceled tick 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 >> v1:172.21.15.71:6800/3911735624 conn(0x7fc58c06e160 legacy=0x7fc58c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 10 monclient: shutdown 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 >> v1:172.21.15.71:6790/0 conn(0x7fc5a006ea40 legacy=0x7fc5a011bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 shutdown_connections 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 >> 172.21.15.71:0/2202124191 conn(0x7fc5a006b630 msgr2=0x7fc5a00b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 shutdown_connections 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 1 -- 172.21.15.71:0/2202124191 wait complete. 2024-03-25T19:46:39.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:39.549+0000 7fc5a5481640 1 librados: shutdown 2024-03-25T19:46:39.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:42.327 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:25.083 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.337 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:25.093 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.356 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:25.112 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.382 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:25.138 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.385 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:25.104 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.390 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:25.109 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.399 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:25.155 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.427 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:25.146 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.433 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:25.189 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.369 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:46.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.486+0000 7f8257686640 10 monclient: get_monmap_and_config 2024-03-25T19:46:46.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.486+0000 7f8257686640 10 monclient: build_initial_monmap 2024-03-25T19:46:46.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.486+0000 7f8257686640 10 monclient: monmap: 2024-03-25T19:46:46.486 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:46:46.487563+0000 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:46:46.487563+0000 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:46:46.487 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:46.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.487+0000 7f8257686640 1 Processor -- start 2024-03-25T19:46:46.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.487+0000 7f8257686640 1 -- start start 2024-03-25T19:46:46.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.487+0000 7f8257686640 10 monclient: init 2024-03-25T19:46:46.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:46.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:46.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 10 monclient(hunting): picked mon.noname-b con 0x7f82500a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:46.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 10 monclient(hunting): picked mon.noname-c con 0x7f825013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:46.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 10 monclient(hunting): picked mon.noname-a con 0x7f82501430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:46.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8250142520 con 0x7f82501430c0 2024-03-25T19:46:46.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500b28a0 con 0x7f825013f710 2024-03-25T19:46:46.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500b2a30 con 0x7f82500a98f0 2024-03-25T19:46:46.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:46.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.488+0000 7f8257686640 10 monclient(hunting): authenticate will time out at 5212.800363s 2024-03-25T19:46:46.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.498+0000 7f8255c23640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f82501430c0 0x7f825014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36888/0 (socket says 172.21.15.71:36888) 2024-03-25T19:46:46.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.498+0000 7f8255c23640 1 -- 172.21.15.71:0/3328861729 learned_addr learned my addr 172.21.15.71:0/3328861729 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:46.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.512+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488707071 0 0) 0x7f8250142520 con 0x7f82501430c0 2024-03-25T19:46:46.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.512+0000 7f8247fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.512+0000 7f8247fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.512+0000 7f8247fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.512+0000 7f8247fff640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:46:46.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.512+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823c003520 con 0x7f82501430c0 2024-03-25T19:46:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1189754417 0 0) 0x7f823c003520 con 0x7f82501430c0 2024-03-25T19:46:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 >> v1:172.21.15.71:6790/0 conn(0x7f82500a98f0 legacy=0x7f82500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 >> v1:172.21.15.193:6789/0 conn(0x7f825013f710 legacy=0x7f82501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:46:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 1 monclient: found mon.noname-a 2024-03-25T19:46:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:46.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82500b44c0 con 0x7f82501430c0 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 10 monclient: _finish_auth 0 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8248001a10 con 0x7f82501430c0 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8257686640 5 monclient: authenticate success, global_id 4500 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8257686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8247fff640 10 monclient: dump: 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:46.525 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:46.526 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:46.526 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:46.526 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:46.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.524+0000 7f8257686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:46.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.543+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8248001be0 con 0x7f82501430c0 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.543+0000 7f8247fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.543+0000 7f8247fff640 1 -- 172.21.15.71:0/3328861729 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8248002030 con 0x7f82501430c0 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.543+0000 7f8247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.543+0000 7f8247fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.543+0000 7f8247fff640 10 monclient: dump: 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:46.543 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:46.544 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:46.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.543+0000 7f8257686640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:46.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.543+0000 7f8257686640 4 set_mon_vals no callback set 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.544+0000 7f8257686640 10 monclient: shutdown 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.544+0000 7f8257686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.544+0000 7f8257686640 1 -- 172.21.15.71:0/3328861729 >> v1:172.21.15.71:6789/0 conn(0x7f82501430c0 legacy=0x7f825014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.544+0000 7f8257686640 1 -- 172.21.15.71:0/3328861729 shutdown_connections 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.544+0000 7f8257686640 1 -- 172.21.15.71:0/3328861729 >> 172.21.15.71:0/3328861729 conn(0x7f82500b7ea0 msgr2=0x7f82500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.544+0000 7f8257686640 1 -- 172.21.15.71:0/3328861729 shutdown_connections 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.544+0000 7f8257686640 1 -- 172.21.15.71:0/3328861729 wait complete. 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 10 monclient: build_initial_monmap 2024-03-25T19:46:46.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 10 monclient: monmap: 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 1 librados: starting msgr at 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 1 librados: starting objecter 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 1 Processor -- start 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 1 -- start start 2024-03-25T19:46:46.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 1 librados: setting wanted keys 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 1 librados: calling monclient init 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.545+0000 7f8257686640 10 monclient: init 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 10 monclient(hunting): picked mon.noname-c con 0x7f82500a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 10 monclient(hunting): picked mon.noname-b con 0x7f825013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 10 monclient(hunting): picked mon.noname-a con 0x7f82501430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825013b280 con 0x7f82501430c0 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8250139080 con 0x7f825013f710 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825013a5e0 con 0x7f82500a98f0 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:46.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.546+0000 7f8257686640 10 monclient(hunting): authenticate will time out at 5212.858044s 2024-03-25T19:46:46.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.562+0000 7f8255422640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f82500a98f0 0x7f8250139ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49166/0 (socket says 172.21.15.71:49166) 2024-03-25T19:46:46.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.562+0000 7f8255422640 1 -- 172.21.15.71:0/36387865 learned_addr learned my addr 172.21.15.71:0/36387865 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:46.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.563+0000 7f8245ffb640 10 client.?.objecter ms_handle_connect 0x7f82500a98f0 2024-03-25T19:46:46.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.563+0000 7f8245ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:46.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.569+0000 7f8245ffb640 10 client.?.objecter ms_handle_connect 0x7f825013f710 2024-03-25T19:46:46.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.569+0000 7f8245ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370009723 0 0) 0x7f8250139080 con 0x7f825013f710 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8234004eb0 con 0x7f825013f710 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 client.?.objecter ms_handle_connect 0x7f82501430c0 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:46.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865219396 0 0) 0x7f825013a5e0 con 0x7f82500a98f0 2024-03-25T19:46:46.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 10 monclient(hunting): my global_id is 14504 2024-03-25T19:46:46.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.578+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8250139080 con 0x7f82500a98f0 2024-03-25T19:46:46.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.593+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3120294553 0 0) 0x7f8250139080 con 0x7f82500a98f0 2024-03-25T19:46:46.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.594+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f825013a5e0 con 0x7f82500a98f0 2024-03-25T19:46:46.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.594+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f824c0019b0 con 0x7f82500a98f0 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.594+0000 7f8245ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.594+0000 7f8245ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.594+0000 7f8245ffb640 10 monclient(hunting): dump: 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:46.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.594+0000 7f8245ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.598+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527908461 0 0) 0x7f8234004eb0 con 0x7f825013f710 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.599+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8250139080 con 0x7f825013f710 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.599+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8240001510 con 0x7f825013f710 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.599+0000 7f8245ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.599+0000 7f8245ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.599+0000 7f8245ffb640 10 monclient(hunting): dump: 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.599 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.600 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:46.600 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:46.600 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:46.600 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:46.600 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:46.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.603+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574336152 0 0) 0x7f825013b280 con 0x7f82501430c0 2024-03-25T19:46:46.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.604+0000 7f8245ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.604+0000 7f8245ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.604+0000 7f8245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.604+0000 7f8245ffb640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:46:46.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.604+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8234004eb0 con 0x7f82501430c0 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2564992237 0 0) 0x7f825013a5e0 con 0x7f82500a98f0 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 >> v1:172.21.15.193:6789/0 conn(0x7f825013f710 legacy=0x7f825013bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 >> v1:172.21.15.71:6789/0 conn(0x7f82501430c0 legacy=0x7f825013c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 1 monclient: found mon.c 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825013b4d0 con 0x7f82500a98f0 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8245ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:46.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8257686640 5 monclient: authenticate success, global_id 14504 2024-03-25T19:46:46.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8257686640 10 monclient: _renew_subs 2024-03-25T19:46:46.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8257686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:46.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.608+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825013b7b0 con 0x7f82500a98f0 2024-03-25T19:46:46.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 10 client.14504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:46.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 10 monclient: _renew_subs 2024-03-25T19:46:46.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:46.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f825013c9f0 con 0x7f82500a98f0 2024-03-25T19:46:46.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 1 librados: init done 2024-03-25T19:46:46.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:46.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:46.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:46.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.609+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825014f2d0 con 0x7f82500a98f0 2024-03-25T19:46:46.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.928+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f824c003610 con 0x7f82500a98f0 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.929+0000 7f8245ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.929+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f824c0039a0 con 0x7f82500a98f0 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.929+0000 7f8245ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.929+0000 7f8245ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.929+0000 7f8245ffb640 10 monclient: dump: 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:46.929 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:46.930 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:46.930 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:46.930 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:46.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.929+0000 7f82477fe640 4 set_mon_vals no callback set 2024-03-25T19:46:46.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.930+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f824c003bc0 con 0x7f82500a98f0 2024-03-25T19:46:46.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.930+0000 7f8254c21640 10 monclient: get_auth_request con 0x7f823406df40 auth_method 0 2024-03-25T19:46:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.930+0000 7f8245ffb640 10 client.14504.objecter ms_handle_connect 0x7f823406df40 2024-03-25T19:46:46.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.936+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f824c054830 con 0x7f82500a98f0 2024-03-25T19:46:46.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.936+0000 7f8245ffb640 10 client.14504.objecter ms_dispatch 0x7f8250000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:46.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.936+0000 7f8245ffb640 3 client.14504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:46.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.936+0000 7f8245ffb640 3 client.14504.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:46.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.936+0000 7f8245ffb640 20 client.14504.objecter dump_active .. 0 homeless 2024-03-25T19:46:46.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.936+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f824c050340 con 0x7f82500a98f0 2024-03-25T19:46:46.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.936+0000 7f8245ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:46.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:46.936+0000 7f8245ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:47.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.160+0000 7f8257686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:47.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.160+0000 7f8257686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:47.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.160+0000 7f8257686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:46:47.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.160+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82500627b0 con 0x7f82500a98f0 2024-03-25T19:46:47.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.236+0000 7f8245ffb640 1 -- 172.21.15.71:0/36387865 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f824c052270 con 0x7f82500a98f0 2024-03-25T19:46:47.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.236+0000 7f8245ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:47.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.236+0000 7f8245ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:47.236 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:46:47.236 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:47.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 10 librados: watch_flush enter 2024-03-25T19:46:47.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 10 librados: watch_flush exit 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 20 client.14504.objecter shutdown clearing up homeless session... 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 10 client.14504.objecter successfully canceled tick 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 >> v1:172.21.15.71:6800/3911735624 conn(0x7f823406df40 legacy=0x7f8234070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 10 monclient: shutdown 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 >> v1:172.21.15.71:6790/0 conn(0x7f82500a98f0 legacy=0x7f8250139ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 shutdown_connections 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 >> 172.21.15.71:0/36387865 conn(0x7f82500b7ea0 msgr2=0x7f8250074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 shutdown_connections 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 1 -- 172.21.15.71:0/36387865 wait complete. 2024-03-25T19:46:47.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:47.239+0000 7f8257686640 1 librados: shutdown 2024-03-25T19:46:47.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:54.057 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:54.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.175+0000 7fe3ff96d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:54.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.175+0000 7fe3ff96d640 10 monclient: build_initial_monmap 2024-03-25T19:46:54.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.175+0000 7fe3ff96d640 10 monclient: monmap: 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:46:54.176815+0000 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:46:54.176815+0000 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:46:54.176 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:54.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.176+0000 7fe3ff96d640 1 Processor -- start 2024-03-25T19:46:54.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 1 -- start start 2024-03-25T19:46:54.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 10 monclient: init 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3f806c360 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3f806d5e0 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3f80af9a0 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f80add00 con 0x7fe3f80af9a0 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f80adfe0 con 0x7fe3f806d5e0 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f806d190 con 0x7fe3f806c360 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:54.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.177+0000 7fe3ff96d640 10 monclient(hunting): authenticate will time out at 5220.489523s 2024-03-25T19:46:54.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.185+0000 7fe3fcf08640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fe3f806d5e0 0x7fe3f80ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:48166/0 (socket says 172.21.15.71:48166) 2024-03-25T19:46:54.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.185+0000 7fe3fcf08640 1 -- 172.21.15.71:0/605463221 learned_addr learned my addr 172.21.15.71:0/605463221 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:54.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.194+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652717103 0 0) 0x7fe3f80adfe0 con 0x7fe3f806d5e0 2024-03-25T19:46:54.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.194+0000 7fe3effff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:54.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.194+0000 7fe3effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:54.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.194+0000 7fe3effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:54.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.194+0000 7fe3effff640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:46:54.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.194+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e4003520 con 0x7fe3f806d5e0 2024-03-25T19:46:54.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2479574187 0 0) 0x7fe3e4003520 con 0x7fe3f806d5e0 2024-03-25T19:46:54.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 >> v1:172.21.15.71:6790/0 conn(0x7fe3f806c360 legacy=0x7fe3f80b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 >> v1:172.21.15.71:6789/0 conn(0x7fe3f80af9a0 legacy=0x7fe3f80aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:54.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 1 monclient: found mon.noname-c 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f80b0d10 con 0x7fe3f806d5e0 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 10 monclient: _finish_auth 0 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3ff96d640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe3e8001690 con 0x7fe3f806d5e0 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3effff640 10 monclient: dump: 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.205 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.206 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:54.206 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:54.206 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:54.206 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:54.206 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:54.206 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:54.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.204+0000 7fe3ff96d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:54.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.217+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3e80017f0 con 0x7fe3f806d5e0 2024-03-25T19:46:54.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.217+0000 7fe3effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:54.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.217+0000 7fe3effff640 1 -- 172.21.15.71:0/605463221 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe3e8001b80 con 0x7fe3f806d5e0 2024-03-25T19:46:54.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.217+0000 7fe3ff96d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:54.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.217+0000 7fe3ff96d640 4 set_mon_vals no callback set 2024-03-25T19:46:54.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.217+0000 7fe3effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:54.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.217+0000 7fe3effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.217+0000 7fe3effff640 10 monclient: dump: 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:54.218 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:54.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.218+0000 7fe3ff96d640 10 monclient: shutdown 2024-03-25T19:46:54.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.218+0000 7fe3ff96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:54.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.218+0000 7fe3ff96d640 1 -- 172.21.15.71:0/605463221 >> v1:172.21.15.193:6789/0 conn(0x7fe3f806d5e0 legacy=0x7fe3f80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.218+0000 7fe3ff96d640 1 -- 172.21.15.71:0/605463221 shutdown_connections 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.218+0000 7fe3ff96d640 1 -- 172.21.15.71:0/605463221 >> 172.21.15.71:0/605463221 conn(0x7fe3f80b5c00 msgr2=0x7fe3f80b6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.218+0000 7fe3ff96d640 1 -- 172.21.15.71:0/605463221 shutdown_connections 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.218+0000 7fe3ff96d640 1 -- 172.21.15.71:0/605463221 wait complete. 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.219+0000 7fe3ff96d640 10 monclient: build_initial_monmap 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.219+0000 7fe3ff96d640 10 monclient: monmap: 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:54.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.219+0000 7fe3ff96d640 1 librados: starting msgr at 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.219+0000 7fe3ff96d640 1 librados: starting objecter 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 1 Processor -- start 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 1 -- start start 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 1 librados: setting wanted keys 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 1 librados: calling monclient init 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 10 monclient: init 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3f806c360 addr v1:172.21.15.193:6789/0 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3f806d5e0 addr v1:172.21.15.71:6790/0 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3f80af9a0 addr v1:172.21.15.71:6789/0 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f8144f10 con 0x7fe3f80af9a0 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f81451f0 con 0x7fe3f806c360 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f8143160 con 0x7fe3f806d5e0 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:54.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.220+0000 7fe3ff96d640 10 monclient(hunting): authenticate will time out at 5220.532542s 2024-03-25T19:46:54.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.235+0000 7fe3fcf08640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe3f806d5e0 0x7fe3f8141030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:54156/0 (socket says 172.21.15.71:54156) 2024-03-25T19:46:54.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.235+0000 7fe3fcf08640 1 -- 172.21.15.71:0/1963082766 learned_addr learned my addr 172.21.15.71:0/1963082766 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:46:54.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.237+0000 7fe3edffb640 10 client.?.objecter ms_handle_connect 0x7fe3f806d5e0 2024-03-25T19:46:54.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.237+0000 7fe3edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:54.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.239+0000 7fe3edffb640 10 client.?.objecter ms_handle_connect 0x7fe3f806c360 2024-03-25T19:46:54.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.239+0000 7fe3edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:54.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.245+0000 7fe3edffb640 10 client.?.objecter ms_handle_connect 0x7fe3f80af9a0 2024-03-25T19:46:54.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.245+0000 7fe3edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:54.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.256+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419202211 0 0) 0x7fe3f81451f0 con 0x7fe3f806c360 2024-03-25T19:46:54.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.256+0000 7fe3edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:54.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.256+0000 7fe3edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:54.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.256+0000 7fe3edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:54.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.256+0000 7fe3edffb640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:46:54.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.256+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc004eb0 con 0x7fe3f806c360 2024-03-25T19:46:54.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.261+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485385506 0 0) 0x7fe3f8144f10 con 0x7fe3f80af9a0 2024-03-25T19:46:54.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.261+0000 7fe3edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:54.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.261+0000 7fe3edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:54.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.261+0000 7fe3edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:54.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.261+0000 7fe3edffb640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:46:54.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.261+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f81451f0 con 0x7fe3f80af9a0 2024-03-25T19:46:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.264+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787223916 0 0) 0x7fe3f8143160 con 0x7fe3f806d5e0 2024-03-25T19:46:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.264+0000 7fe3edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.264+0000 7fe3edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.264+0000 7fe3edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.264+0000 7fe3edffb640 10 monclient(hunting): my global_id is 14510 2024-03-25T19:46:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.264+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f8144f10 con 0x7fe3f806d5e0 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.275+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2380015686 0 0) 0x7fe3dc004eb0 con 0x7fe3f806c360 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.275+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3f8143160 con 0x7fe3f806c360 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.275+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe3f4001930 con 0x7fe3f806c360 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.275+0000 7fe3edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.275+0000 7fe3edffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.275+0000 7fe3edffb640 10 monclient(hunting): dump: 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:54.275 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:54.276 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:54.276 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:54.276 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:54.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.275+0000 7fe3edffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:54.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.277+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3167025357 0 0) 0x7fe3f8144f10 con 0x7fe3f806d5e0 2024-03-25T19:46:54.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.278+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3dc004eb0 con 0x7fe3f806d5e0 2024-03-25T19:46:54.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.278+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe3e8001b00 con 0x7fe3f806d5e0 2024-03-25T19:46:54.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.278+0000 7fe3edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.278+0000 7fe3edffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.278+0000 7fe3edffb640 10 monclient(hunting): dump: 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:54.278 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.283+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2285901938 0 0) 0x7fe3f81451f0 con 0x7fe3f80af9a0 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.284+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3f8144f10 con 0x7fe3f80af9a0 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.284+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe3f0001b10 con 0x7fe3f80af9a0 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.284+0000 7fe3edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.284+0000 7fe3edffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.284+0000 7fe3edffb640 10 monclient(hunting): dump: 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:54.284 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1990169663 0 0) 0x7fe3f8144f10 con 0x7fe3f80af9a0 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 >> v1:172.21.15.71:6790/0 conn(0x7fe3f806d5e0 legacy=0x7fe3f8141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 >> v1:172.21.15.193:6789/0 conn(0x7fe3f806c360 legacy=0x7fe3f8140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 1 monclient: found mon.a 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8146690 con 0x7fe3f80af9a0 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.307+0000 7fe3ff96d640 5 monclient: authenticate success, global_id 4509 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.308+0000 7fe3ff96d640 10 monclient: _renew_subs 2024-03-25T19:46:54.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.308+0000 7fe3ff96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:46:54.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.308+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f8142b90 con 0x7fe3f80af9a0 2024-03-25T19:46:54.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.308+0000 7fe3ff96d640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:54.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.308+0000 7fe3ff96d640 10 monclient: _renew_subs 2024-03-25T19:46:54.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.308+0000 7fe3ff96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:46:54.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.308+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f8143a60 con 0x7fe3f80af9a0 2024-03-25T19:46:54.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.308+0000 7fe3ff96d640 1 librados: init done 2024-03-25T19:46:54.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.309+0000 7fe3ff96d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:54.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.309+0000 7fe3ff96d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:54.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.309+0000 7fe3ff96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:46:54.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.309+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f81432f0 con 0x7fe3f80af9a0 2024-03-25T19:46:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.635+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3f0001fe0 con 0x7fe3f80af9a0 2024-03-25T19:46:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.635+0000 7fe3edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.635+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fe3f0003b10 con 0x7fe3f80af9a0 2024-03-25T19:46:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.635+0000 7fe3edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.635+0000 7fe3edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.635+0000 7fe3edffb640 10 monclient: dump: 2024-03-25T19:46:54.635 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:46:54.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.635+0000 7fe3ef7fe640 4 set_mon_vals no callback set 2024-03-25T19:46:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.636+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fe3f0003d30 con 0x7fe3f80af9a0 2024-03-25T19:46:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.637+0000 7fe3fd709640 10 monclient: get_auth_request con 0x7fe3dc06e110 auth_method 0 2024-03-25T19:46:54.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.637+0000 7fe3edffb640 10 client.4509.objecter ms_handle_connect 0x7fe3dc06e110 2024-03-25T19:46:54.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.642+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fe3f0053830 con 0x7fe3f80af9a0 2024-03-25T19:46:54.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.642+0000 7fe3edffb640 10 client.4509.objecter ms_dispatch 0x7fe3f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.642+0000 7fe3edffb640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.642+0000 7fe3edffb640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.642+0000 7fe3edffb640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T19:46:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.642+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3f004f340 con 0x7fe3f80af9a0 2024-03-25T19:46:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.642+0000 7fe3edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.642+0000 7fe3edffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:54.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.870+0000 7fe3ff96d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:54.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.870+0000 7fe3ff96d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:54.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.870+0000 7fe3ff96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:46:54.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.870+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3f806c740 con 0x7fe3f80af9a0 2024-03-25T19:46:54.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.913+0000 7fe3edffb640 1 -- 172.21.15.71:0/1963082766 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fe3f0051270 con 0x7fe3f80af9a0 2024-03-25T19:46:54.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.913+0000 7fe3edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:54.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.913+0000 7fe3edffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:54.913 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:46:54.914 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.915+0000 7fe3ff96d640 10 librados: watch_flush enter 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.915+0000 7fe3ff96d640 10 librados: watch_flush exit 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.915+0000 7fe3ff96d640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.915+0000 7fe3ff96d640 10 client.4509.objecter successfully canceled tick 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 >> v1:172.21.15.71:6800/3911735624 conn(0x7fe3dc06e110 legacy=0x7fe3dc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 10 monclient: shutdown 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 >> v1:172.21.15.71:6789/0 conn(0x7fe3f80af9a0 legacy=0x7fe3f81e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 shutdown_connections 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 >> 172.21.15.71:0/1963082766 conn(0x7fe3f80b5c00 msgr2=0x7fe3f813fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 shutdown_connections 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 1 -- 172.21.15.71:0/1963082766 wait complete. 2024-03-25T19:46:54.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:46:54.916+0000 7fe3ff96d640 1 librados: shutdown 2024-03-25T19:46:54.927 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:01.733 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:01.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.850+0000 7f61968b3640 10 monclient: get_monmap_and_config 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.850+0000 7f61968b3640 10 monclient: build_initial_monmap 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.850+0000 7f61968b3640 10 monclient: monmap: 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:47:01.850953+0000 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:47:01.850953+0000 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:47:01.850 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:47:01.851 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.851+0000 7f61968b3640 1 Processor -- start 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.851+0000 7f61968b3640 1 -- start start 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.851+0000 7f61968b3640 10 monclient: init 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 10 monclient(hunting): picked mon.noname-b con 0x7f61900a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 10 monclient(hunting): picked mon.noname-a con 0x7f619013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 10 monclient(hunting): picked mon.noname-c con 0x7f61901430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6190142520 con 0x7f619013f710 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61900b28a0 con 0x7f61901430c0 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61900b2a30 con 0x7f61900a98f0 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:01.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.852+0000 7f61968b3640 10 monclient(hunting): authenticate will time out at 5228.164115s 2024-03-25T19:47:01.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.861+0000 7f618ffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f61900a98f0 0x7f61900a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33888/0 (socket says 172.21.15.71:33888) 2024-03-25T19:47:01.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.862+0000 7f618ffff640 1 -- 172.21.15.71:0/2196753367 learned_addr learned my addr 172.21.15.71:0/2196753367 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:01.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.875+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138554648 0 0) 0x7f61900b28a0 con 0x7f61901430c0 2024-03-25T19:47:01.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.875+0000 7f618effd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:01.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.875+0000 7f618effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:01.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.875+0000 7f618effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:01.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.875+0000 7f618effd640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:47:01.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.875+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6178003520 con 0x7f61901430c0 2024-03-25T19:47:01.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.878+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532251674 0 0) 0x7f61900b2a30 con 0x7f61900a98f0 2024-03-25T19:47:01.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.878+0000 7f618effd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:01.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.878+0000 7f618effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:01.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.878+0000 7f618effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:01.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.878+0000 7f618effd640 10 monclient(hunting): my global_id is 14513 2024-03-25T19:47:01.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.878+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61900b28a0 con 0x7f61900a98f0 2024-03-25T19:47:01.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.881+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685163565 0 0) 0x7f6190142520 con 0x7f619013f710 2024-03-25T19:47:01.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.881+0000 7f618effd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:01.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.881+0000 7f618effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:01.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.881+0000 7f618effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:01.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.881+0000 7f618effd640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:47:01.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.881+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61900b2a30 con 0x7f619013f710 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 475031159 0 0) 0x7f6178003520 con 0x7f61901430c0 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 >> v1:172.21.15.71:6790/0 conn(0x7f61900a98f0 legacy=0x7f61900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 >> v1:172.21.15.71:6789/0 conn(0x7f619013f710 legacy=0x7f61901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 1 monclient: found mon.noname-c 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61900b44c0 con 0x7f61901430c0 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 10 monclient: _finish_auth 0 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6180001690 con 0x7f61901430c0 2024-03-25T19:47:01.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f618effd640 10 monclient: dump: 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f61968b3640 5 monclient: authenticate success, global_id 4507 2024-03-25T19:47:01.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.884+0000 7f61968b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.895+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61800017f0 con 0x7f61901430c0 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.895+0000 7f618effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.895+0000 7f618effd640 1 -- 172.21.15.71:0/2196753367 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6180001c00 con 0x7f61901430c0 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.895+0000 7f61968b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.895+0000 7f61968b3640 4 set_mon_vals no callback set 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.895+0000 7f618effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.895+0000 7f618effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.895+0000 7f618effd640 10 monclient: dump: 2024-03-25T19:47:01.895 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:01.896 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:01.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.896+0000 7f61968b3640 10 monclient: shutdown 2024-03-25T19:47:01.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.896+0000 7f61968b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:01.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.896+0000 7f61968b3640 1 -- 172.21.15.71:0/2196753367 >> v1:172.21.15.193:6789/0 conn(0x7f61901430c0 legacy=0x7f619014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:01.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.896+0000 7f61968b3640 1 -- 172.21.15.71:0/2196753367 shutdown_connections 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.896+0000 7f61968b3640 1 -- 172.21.15.71:0/2196753367 >> 172.21.15.71:0/2196753367 conn(0x7f61900b7ea0 msgr2=0x7f61900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.896+0000 7f61968b3640 1 -- 172.21.15.71:0/2196753367 shutdown_connections 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.896+0000 7f61968b3640 1 -- 172.21.15.71:0/2196753367 wait complete. 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.897+0000 7f61968b3640 10 monclient: build_initial_monmap 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.897+0000 7f61968b3640 10 monclient: monmap: 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:47:01.898 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:47:01.899 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:01.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.897+0000 7f61968b3640 1 librados: starting msgr at 2024-03-25T19:47:01.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.897+0000 7f61968b3640 1 librados: starting objecter 2024-03-25T19:47:01.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.897+0000 7f61968b3640 1 Processor -- start 2024-03-25T19:47:01.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 1 -- start start 2024-03-25T19:47:01.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 1 librados: setting wanted keys 2024-03-25T19:47:01.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 1 librados: calling monclient init 2024-03-25T19:47:01.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 10 monclient: init 2024-03-25T19:47:01.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:01.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:01.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 10 monclient(hunting): picked mon.noname-b con 0x7f61900a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:01.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 10 monclient(hunting): picked mon.noname-a con 0x7f619013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:01.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 10 monclient(hunting): picked mon.noname-c con 0x7f61901430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:01.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61901db170 con 0x7f619013f710 2024-03-25T19:47:01.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61901dac50 con 0x7f61900a98f0 2024-03-25T19:47:01.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61901dade0 con 0x7f61901430c0 2024-03-25T19:47:01.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:01.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.898+0000 7f61968b3640 10 monclient(hunting): authenticate will time out at 5228.210736s 2024-03-25T19:47:01.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.914+0000 7f618ffff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f61900a98f0 0x7f61901394b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:55354/0 (socket says 172.21.15.71:55354) 2024-03-25T19:47:01.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.914+0000 7f618ffff640 1 -- 172.21.15.71:0/308912160 learned_addr learned my addr 172.21.15.71:0/308912160 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:01.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.916+0000 7f618cff9640 10 client.?.objecter ms_handle_connect 0x7f61900a98f0 2024-03-25T19:47:01.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.916+0000 7f618cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:01.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.921+0000 7f618cff9640 10 client.?.objecter ms_handle_connect 0x7f61901430c0 2024-03-25T19:47:01.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.921+0000 7f618cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:01.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.924+0000 7f618cff9640 10 client.?.objecter ms_handle_connect 0x7f619013f710 2024-03-25T19:47:01.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.924+0000 7f618cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:01.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.927+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565388433 0 0) 0x7f61901dac50 con 0x7f61900a98f0 2024-03-25T19:47:01.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.927+0000 7f618cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:01.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.927+0000 7f618cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:01.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.927+0000 7f618cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:01.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.927+0000 7f618cff9640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:47:01.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.927+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617c004eb0 con 0x7f61900a98f0 2024-03-25T19:47:01.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.936+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156352628 0 0) 0x7f61901dade0 con 0x7f61901430c0 2024-03-25T19:47:01.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.936+0000 7f618cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:01.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.936+0000 7f618cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:01.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.936+0000 7f618cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:01.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.936+0000 7f618cff9640 10 monclient(hunting): my global_id is 14516 2024-03-25T19:47:01.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.936+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61901dac50 con 0x7f61901430c0 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2383974891 0 0) 0x7f617c004eb0 con 0x7f61900a98f0 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61901dade0 con 0x7f61900a98f0 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6188001d70 con 0x7f61900a98f0 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 10 monclient(hunting): dump: 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:01.938 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365143400 0 0) 0x7f61901db170 con 0x7f619013f710 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:47:01.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.938+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617c004eb0 con 0x7f619013f710 2024-03-25T19:47:01.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 123590780 0 0) 0x7f61901dade0 con 0x7f61900a98f0 2024-03-25T19:47:01.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 >> v1:172.21.15.71:6790/0 conn(0x7f61901430c0 legacy=0x7f6190132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:01.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 >> v1:172.21.15.71:6789/0 conn(0x7f619013f710 legacy=0x7f61901d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:01.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:01.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 1 monclient: found mon.b 2024-03-25T19:47:01.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619013c170 con 0x7f61900a98f0 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f618cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 5 monclient: authenticate success, global_id 4510 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 10 monclient: _renew_subs 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619013c420 con 0x7f61900a98f0 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 10 monclient: _renew_subs 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6190133860 con 0x7f61900a98f0 2024-03-25T19:47:01.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.944+0000 7f61968b3640 1 librados: init done 2024-03-25T19:47:01.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.945+0000 7f61968b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:01.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.945+0000 7f61968b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:01.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.945+0000 7f61968b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:01.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:01.945+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61901daf70 con 0x7f61900a98f0 2024-03-25T19:47:02.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.218+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6188001fd0 con 0x7f61900a98f0 2024-03-25T19:47:02.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.218+0000 7f618cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:02.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.218+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6188004fb0 con 0x7f61900a98f0 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.218+0000 7f618cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.218+0000 7f618cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.218+0000 7f618cff9640 10 monclient: dump: 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:02.219 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:02.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.218+0000 7f618e7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:02.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.219+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f618801af80 con 0x7f61900a98f0 2024-03-25T19:47:02.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.220+0000 7f618f7fe640 10 monclient: get_auth_request con 0x7f617c06de00 auth_method 0 2024-03-25T19:47:02.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.220+0000 7f618cff9640 10 client.4510.objecter ms_handle_connect 0x7f617c06de00 2024-03-25T19:47:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.224+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f6188054ba0 con 0x7f61900a98f0 2024-03-25T19:47:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.224+0000 7f618cff9640 10 client.4510.objecter ms_dispatch 0x7f6190000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.224+0000 7f618cff9640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.224+0000 7f618cff9640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.224+0000 7f618cff9640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T19:47:02.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.225+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61880506b0 con 0x7f61900a98f0 2024-03-25T19:47:02.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.225+0000 7f618cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:02.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.225+0000 7f618cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:02.411 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:45.167 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.420 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:45.176 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.452+0000 7f61968b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:02.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.452+0000 7f61968b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:02.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.452+0000 7f61968b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:02.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.452+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61900627b0 con 0x7f61900a98f0 2024-03-25T19:47:02.461 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:06:45.217 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.463 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:45.182 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.469 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:45.226 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.470 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:45.190 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.474 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:45.230 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.513 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:45.232 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.513+0000 7f618cff9640 1 -- 172.21.15.71:0/308912160 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f61880525e0 con 0x7f61900a98f0 2024-03-25T19:47:02.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.513+0000 7f618cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:02.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.513+0000 7f618cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:02.514 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:47:02.514 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 10 librados: watch_flush enter 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 10 librados: watch_flush exit 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 10 client.4510.objecter successfully canceled tick 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 >> v1:172.21.15.71:6800/3911735624 conn(0x7f617c06de00 legacy=0x7f617c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 10 monclient: shutdown 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 >> v1:172.21.15.193:6789/0 conn(0x7f61900a98f0 legacy=0x7f61901394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 shutdown_connections 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.515+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 >> 172.21.15.71:0/308912160 conn(0x7f61900b7ea0 msgr2=0x7f619014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:02.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.516+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 shutdown_connections 2024-03-25T19:47:02.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.516+0000 7f61968b3640 1 -- 172.21.15.71:0/308912160 wait complete. 2024-03-25T19:47:02.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:02.516+0000 7f61968b3640 1 librados: shutdown 2024-03-25T19:47:02.517 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:06:45.272 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:02.527 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:09.333 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:09.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.451+0000 7f493017c640 10 monclient: get_monmap_and_config 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.451+0000 7f493017c640 10 monclient: build_initial_monmap 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.451+0000 7f493017c640 10 monclient: monmap: 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:47:09.452852+0000 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:47:09.452852+0000 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:47:09.452 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:09.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.452+0000 7f493017c640 1 Processor -- start 2024-03-25T19:47:09.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 1 -- start start 2024-03-25T19:47:09.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 10 monclient: init 2024-03-25T19:47:09.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:09.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:09.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 10 monclient(hunting): picked mon.noname-c con 0x7f49280a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:09.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 10 monclient(hunting): picked mon.noname-a con 0x7f492813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:09.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 10 monclient(hunting): picked mon.noname-b con 0x7f49281430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:09.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4928142520 con 0x7f492813f710 2024-03-25T19:47:09.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49280b28a0 con 0x7f49280a98f0 2024-03-25T19:47:09.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49280b2a30 con 0x7f49281430c0 2024-03-25T19:47:09.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:09.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.453+0000 7f493017c640 10 monclient(hunting): authenticate will time out at 5235.765701s 2024-03-25T19:47:09.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.465+0000 7f492df18640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f49280a98f0 0x7f49280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:52916/0 (socket says 172.21.15.71:52916) 2024-03-25T19:47:09.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.465+0000 7f492df18640 1 -- 172.21.15.71:0/2179018116 learned_addr learned my addr 172.21.15.71:0/2179018116 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:09.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.476+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423625719 0 0) 0x7f49280b28a0 con 0x7f49280a98f0 2024-03-25T19:47:09.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.476+0000 7f492cf16640 10 monclient(hunting): _negotiate 2024-03-25T19:47:09.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.476+0000 7f492cf16640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:09.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.476+0000 7f492cf16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:09.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.476+0000 7f492cf16640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:47:09.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.476+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4914003520 con 0x7f49280a98f0 2024-03-25T19:47:09.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.487+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989220586 0 0) 0x7f49280b2a30 con 0x7f49281430c0 2024-03-25T19:47:09.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.487+0000 7f492cf16640 10 monclient(hunting): _negotiate 2024-03-25T19:47:09.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.487+0000 7f492cf16640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:09.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.487+0000 7f492cf16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:09.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.487+0000 7f492cf16640 10 monclient(hunting): my global_id is 14519 2024-03-25T19:47:09.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.487+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49280b28a0 con 0x7f49281430c0 2024-03-25T19:47:09.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332784754 0 0) 0x7f4914003520 con 0x7f49280a98f0 2024-03-25T19:47:09.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 >> v1:172.21.15.71:6790/0 conn(0x7f49281430c0 legacy=0x7f492814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:09.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 >> v1:172.21.15.71:6789/0 conn(0x7f492813f710 legacy=0x7f49281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:09.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 10 monclient: _finish_hunting 0 2024-03-25T19:47:09.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 1 monclient: found mon.noname-c 2024-03-25T19:47:09.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:09.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:47:09.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280b44c0 con 0x7f49280a98f0 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 10 monclient: _finish_auth 0 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 10 monclient: _check_auth_tickets 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4924001690 con 0x7f49280a98f0 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f492cf16640 10 monclient: dump: 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:09.491 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:09.492 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:09.492 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:09.492 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:09.492 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:09.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f493017c640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:47:09.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.490+0000 7f493017c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:09.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49240017f0 con 0x7f49280a98f0 2024-03-25T19:47:09.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f492cf16640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f492cf16640 1 -- 172.21.15.71:0/2179018116 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4924001c00 con 0x7f49280a98f0 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f492cf16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f492cf16640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f492cf16640 10 monclient: dump: 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:09.499 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:09.500 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:09.500 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:09.500 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:09.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f493017c640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:09.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f493017c640 4 set_mon_vals no callback set 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f493017c640 10 monclient: shutdown 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f493017c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f493017c640 1 -- 172.21.15.71:0/2179018116 >> v1:172.21.15.193:6789/0 conn(0x7f49280a98f0 legacy=0x7f49280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.499+0000 7f493017c640 1 -- 172.21.15.71:0/2179018116 shutdown_connections 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.500+0000 7f493017c640 1 -- 172.21.15.71:0/2179018116 >> 172.21.15.71:0/2179018116 conn(0x7f49280b7ea0 msgr2=0x7f49280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.500+0000 7f493017c640 1 -- 172.21.15.71:0/2179018116 shutdown_connections 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.500+0000 7f493017c640 1 -- 172.21.15.71:0/2179018116 wait complete. 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.500+0000 7f493017c640 10 monclient: build_initial_monmap 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.500+0000 7f493017c640 10 monclient: monmap: 2024-03-25T19:47:09.501 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.501+0000 7f493017c640 1 librados: starting msgr at 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.501+0000 7f493017c640 1 librados: starting objecter 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.501+0000 7f493017c640 1 Processor -- start 2024-03-25T19:47:09.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.501+0000 7f493017c640 1 -- start start 2024-03-25T19:47:09.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.501+0000 7f493017c640 1 librados: setting wanted keys 2024-03-25T19:47:09.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.501+0000 7f493017c640 1 librados: calling monclient init 2024-03-25T19:47:09.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.501+0000 7f493017c640 10 monclient: init 2024-03-25T19:47:09.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.501+0000 7f493017c640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 10 monclient(hunting): picked mon.noname-a con 0x7f49280a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 10 monclient(hunting): picked mon.noname-c con 0x7f492813f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 10 monclient(hunting): picked mon.noname-b con 0x7f49281430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4928135590 con 0x7f49280a98f0 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4928136e50 con 0x7f49281430c0 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4928135070 con 0x7f492813f710 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:09.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.502+0000 7f493017c640 10 monclient(hunting): authenticate will time out at 5235.814002s 2024-03-25T19:47:09.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.507+0000 7f492d717640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f492813f710 0x7f49281d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56046/0 (socket says 172.21.15.71:56046) 2024-03-25T19:47:09.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.507+0000 7f492d717640 1 -- 172.21.15.71:0/2834156654 learned_addr learned my addr 172.21.15.71:0/2834156654 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:09.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.515+0000 7f491e7fc640 10 client.?.objecter ms_handle_connect 0x7f492813f710 2024-03-25T19:47:09.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.515+0000 7f491e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:09.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.524+0000 7f491e7fc640 10 client.?.objecter ms_handle_connect 0x7f49280a98f0 2024-03-25T19:47:09.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.524+0000 7f491e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:09.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.527+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303293754 0 0) 0x7f4928135070 con 0x7f492813f710 2024-03-25T19:47:09.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.527+0000 7f491e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:09.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.527+0000 7f491e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:09.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.527+0000 7f491e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:09.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.527+0000 7f491e7fc640 10 monclient(hunting): my global_id is 14522 2024-03-25T19:47:09.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.527+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490c004eb0 con 0x7f492813f710 2024-03-25T19:47:09.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.529+0000 7f491e7fc640 10 client.?.objecter ms_handle_connect 0x7f49281430c0 2024-03-25T19:47:09.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.529+0000 7f491e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:09.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.534+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291667318 0 0) 0x7f4928135590 con 0x7f49280a98f0 2024-03-25T19:47:09.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.534+0000 7f491e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:09.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.534+0000 7f491e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:09.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.534+0000 7f491e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:09.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.534+0000 7f491e7fc640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:47:09.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.535+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4928135070 con 0x7f49280a98f0 2024-03-25T19:47:09.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.541+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553663491 0 0) 0x7f4928136e50 con 0x7f49281430c0 2024-03-25T19:47:09.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.541+0000 7f491e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:09.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.541+0000 7f491e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:09.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.541+0000 7f491e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:09.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.541+0000 7f491e7fc640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:47:09.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.541+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4928135590 con 0x7f49281430c0 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.557+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 855583395 0 0) 0x7f4928135070 con 0x7f49280a98f0 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.557+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4928136e50 con 0x7f49280a98f0 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.557+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4924001b00 con 0x7f49280a98f0 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.557+0000 7f491e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.557+0000 7f491e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.557+0000 7f491e7fc640 10 monclient(hunting): dump: 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:09.557 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:09.558 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:09.558 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:09.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.557+0000 7f491e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:09.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.559+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2126379455 0 0) 0x7f490c004eb0 con 0x7f492813f710 2024-03-25T19:47:09.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.559+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4928135070 con 0x7f492813f710 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.559+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f49180019d0 con 0x7f492813f710 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.559+0000 7f491e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.559+0000 7f491e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.559+0000 7f491e7fc640 10 monclient(hunting): dump: 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:09.559 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:09.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2284639609 0 0) 0x7f4928135070 con 0x7f492813f710 2024-03-25T19:47:09.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 >> v1:172.21.15.193:6789/0 conn(0x7f49281430c0 legacy=0x7f49281e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:09.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 >> v1:172.21.15.71:6789/0 conn(0x7f49280a98f0 legacy=0x7f4928130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:09.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:09.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 1 monclient: found mon.c 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4928135d80 con 0x7f492813f710 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f491e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 5 monclient: authenticate success, global_id 14522 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 10 monclient: _renew_subs 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:09.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4928131660 con 0x7f492813f710 2024-03-25T19:47:09.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 10 client.14522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:09.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 10 monclient: _renew_subs 2024-03-25T19:47:09.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:09.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49280b78a0 con 0x7f492813f710 2024-03-25T19:47:09.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.570+0000 7f493017c640 1 librados: init done 2024-03-25T19:47:09.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.571+0000 7f493017c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:09.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.571+0000 7f493017c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:09.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.571+0000 7f493017c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:09.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.571+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4928135f70 con 0x7f492813f710 2024-03-25T19:47:09.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.902+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4918003610 con 0x7f492813f710 2024-03-25T19:47:09.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.902+0000 7f491e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:09.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.902+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f4918003840 con 0x7f492813f710 2024-03-25T19:47:09.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.902+0000 7f491e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:09.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.902+0000 7f491e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:09.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.902+0000 7f491e7fc640 10 monclient: dump: 2024-03-25T19:47:09.903 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:09.903 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:09.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.902+0000 7f491ffff640 4 set_mon_vals no callback set 2024-03-25T19:47:09.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.903+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f4918003a60 con 0x7f492813f710 2024-03-25T19:47:09.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.904+0000 7f492df18640 10 monclient: get_auth_request con 0x7f490c06dd80 auth_method 0 2024-03-25T19:47:09.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.904+0000 7f491e7fc640 10 client.14522.objecter ms_handle_connect 0x7f490c06dd80 2024-03-25T19:47:09.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.909+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f4918004460 con 0x7f492813f710 2024-03-25T19:47:09.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.909+0000 7f491e7fc640 10 client.14522.objecter ms_dispatch 0x7f4928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:09.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.909+0000 7f491e7fc640 3 client.14522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:09.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.909+0000 7f491e7fc640 3 client.14522.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:09.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.909+0000 7f491e7fc640 20 client.14522.objecter dump_active .. 0 homeless 2024-03-25T19:47:09.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.909+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4918050190 con 0x7f492813f710 2024-03-25T19:47:09.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.909+0000 7f491e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:09.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:09.909+0000 7f491e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:10.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.139+0000 7f493017c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:10.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.139+0000 7f493017c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:10.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.139+0000 7f493017c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:10.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.139+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4928135200 con 0x7f492813f710 2024-03-25T19:47:10.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.183+0000 7f491e7fc640 1 -- 172.21.15.71:0/2834156654 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f49180520c0 con 0x7f492813f710 2024-03-25T19:47:10.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.183+0000 7f491e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:10.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.183+0000 7f491e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:10.183 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:47:10.183 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 10 librados: watch_flush enter 2024-03-25T19:47:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 10 librados: watch_flush exit 2024-03-25T19:47:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 20 client.14522.objecter shutdown clearing up homeless session... 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 10 client.14522.objecter successfully canceled tick 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 >> v1:172.21.15.71:6800/3911735624 conn(0x7f490c06dd80 legacy=0x7f490c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 10 monclient: shutdown 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 >> v1:172.21.15.71:6790/0 conn(0x7f492813f710 legacy=0x7f49281d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 shutdown_connections 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 >> 172.21.15.71:0/2834156654 conn(0x7f49280b7ea0 msgr2=0x7f4928074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 shutdown_connections 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 1 -- 172.21.15.71:0/2834156654 wait complete. 2024-03-25T19:47:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:10.186+0000 7f493017c640 1 librados: shutdown 2024-03-25T19:47:10.197 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:17.003 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.119+0000 7f8c64563640 10 monclient: get_monmap_and_config 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.119+0000 7f8c64563640 10 monclient: build_initial_monmap 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.119+0000 7f8c64563640 10 monclient: monmap: 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:47:17.120229+0000 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:47:17.120229+0000 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:47:17.119 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:47:17.120 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:17.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.120+0000 7f8c64563640 1 Processor -- start 2024-03-25T19:47:17.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.120+0000 7f8c64563640 1 -- start start 2024-03-25T19:47:17.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.120+0000 7f8c64563640 10 monclient: init 2024-03-25T19:47:17.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:17.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:17.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 10 monclient(hunting): picked mon.noname-c con 0x7f8c5c0704a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:17.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 10 monclient(hunting): picked mon.noname-a con 0x7f8c5c070880 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:17.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 10 monclient(hunting): picked mon.noname-b con 0x7f8c5c149a40 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:17.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c143870 con 0x7f8c5c070880 2024-03-25T19:47:17.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c072480 con 0x7f8c5c0704a0 2024-03-25T19:47:17.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c072760 con 0x7f8c5c149a40 2024-03-25T19:47:17.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:17.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.121+0000 7f8c64563640 10 monclient(hunting): authenticate will time out at 5243.433365s 2024-03-25T19:47:17.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.130+0000 7f8c61afe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8c5c070880 0x7f8c5c13d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48022/0 (socket says 172.21.15.71:48022) 2024-03-25T19:47:17.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.130+0000 7f8c61afe640 1 -- 172.21.15.71:0/3503719393 learned_addr learned my addr 172.21.15.71:0/3503719393 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:17.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.144+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191543917 0 0) 0x7f8c5c072480 con 0x7f8c5c0704a0 2024-03-25T19:47:17.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.144+0000 7f8c612fd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:17.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.144+0000 7f8c612fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:17.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.144+0000 7f8c612fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:17.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.144+0000 7f8c612fd640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:47:17.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.144+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c48003520 con 0x7f8c5c0704a0 2024-03-25T19:47:17.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.151+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006634836 0 0) 0x7f8c5c143870 con 0x7f8c5c070880 2024-03-25T19:47:17.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.151+0000 7f8c612fd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:17.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.151+0000 7f8c612fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:17.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.151+0000 7f8c612fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:17.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.151+0000 7f8c612fd640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:47:17.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.151+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5c072480 con 0x7f8c5c070880 2024-03-25T19:47:17.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.162+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134296397 0 0) 0x7f8c5c072760 con 0x7f8c5c149a40 2024-03-25T19:47:17.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.162+0000 7f8c612fd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:17.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.162+0000 7f8c612fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:17.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.162+0000 7f8c612fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:17.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.162+0000 7f8c612fd640 10 monclient(hunting): my global_id is 14525 2024-03-25T19:47:17.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.162+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5c143870 con 0x7f8c5c149a40 2024-03-25T19:47:17.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655553371 0 0) 0x7f8c5c072480 con 0x7f8c5c070880 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 >> v1:172.21.15.71:6790/0 conn(0x7f8c5c149a40 legacy=0x7f8c5c14be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 >> v1:172.21.15.193:6789/0 conn(0x7f8c5c0704a0 legacy=0x7f8c5c143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 1 monclient: found mon.noname-a 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c072a40 con 0x7f8c5c070880 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 10 monclient: _finish_auth 0 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8c4c001670 con 0x7f8c5c070880 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:17.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c612fd640 10 monclient: dump: 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c64563640 5 monclient: authenticate success, global_id 4524 2024-03-25T19:47:17.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.173+0000 7f8c64563640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.189+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c4c0017d0 con 0x7f8c5c070880 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.189+0000 7f8c612fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.189+0000 7f8c612fd640 1 -- 172.21.15.71:0/3503719393 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8c4c001b60 con 0x7f8c5c070880 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.189+0000 7f8c64563640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.189+0000 7f8c64563640 4 set_mon_vals no callback set 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.189+0000 7f8c612fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.189+0000 7f8c612fd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.189+0000 7f8c612fd640 10 monclient: dump: 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:17.189 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:17.190 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:17.190 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:17.190 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:17.190 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:17.190 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:17.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.190+0000 7f8c64563640 10 monclient: shutdown 2024-03-25T19:47:17.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.190+0000 7f8c64563640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:17.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.190+0000 7f8c64563640 1 -- 172.21.15.71:0/3503719393 >> v1:172.21.15.71:6789/0 conn(0x7f8c5c070880 legacy=0x7f8c5c13d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:17.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.190+0000 7f8c64563640 1 -- 172.21.15.71:0/3503719393 shutdown_connections 2024-03-25T19:47:17.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.190+0000 7f8c64563640 1 -- 172.21.15.71:0/3503719393 >> 172.21.15.71:0/3503719393 conn(0x7f8c5c0b5c00 msgr2=0x7f8c5c0b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:17.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.190+0000 7f8c64563640 1 -- 172.21.15.71:0/3503719393 shutdown_connections 2024-03-25T19:47:17.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.190+0000 7f8c64563640 1 -- 172.21.15.71:0/3503719393 wait complete. 2024-03-25T19:47:17.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 10 monclient: build_initial_monmap 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 10 monclient: monmap: 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 1 librados: starting msgr at 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 1 librados: starting objecter 2024-03-25T19:47:17.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 1 Processor -- start 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 1 -- start start 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 1 librados: setting wanted keys 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 1 librados: calling monclient init 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.191+0000 7f8c64563640 10 monclient: init 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 10 monclient(hunting): picked mon.noname-a con 0x7f8c5c0704a0 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 10 monclient(hunting): picked mon.noname-c con 0x7f8c5c070880 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 10 monclient(hunting): picked mon.noname-b con 0x7f8c5c149a40 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c13a230 con 0x7f8c5c0704a0 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c132740 con 0x7f8c5c149a40 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c13cea0 con 0x7f8c5c070880 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:17.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.192+0000 7f8c64563640 10 monclient(hunting): authenticate will time out at 5243.504320s 2024-03-25T19:47:17.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.196+0000 7f8c61afe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8c5c070880 0x7f8c5c131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42110/0 (socket says 172.21.15.71:42110) 2024-03-25T19:47:17.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.196+0000 7f8c61afe640 1 -- 172.21.15.71:0/1029608915 learned_addr learned my addr 172.21.15.71:0/1029608915 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:17.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.203+0000 7f8c52ffd640 10 client.?.objecter ms_handle_connect 0x7f8c5c070880 2024-03-25T19:47:17.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.203+0000 7f8c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:17.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.215+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644352438 0 0) 0x7f8c5c13cea0 con 0x7f8c5c070880 2024-03-25T19:47:17.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.215+0000 7f8c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:17.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.215+0000 7f8c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:17.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.215+0000 7f8c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:17.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.215+0000 7f8c52ffd640 10 monclient(hunting): my global_id is 14528 2024-03-25T19:47:17.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.215+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c40004eb0 con 0x7f8c5c070880 2024-03-25T19:47:17.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.221+0000 7f8c52ffd640 10 client.?.objecter ms_handle_connect 0x7f8c5c0704a0 2024-03-25T19:47:17.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.222+0000 7f8c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:17.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2539871954 0 0) 0x7f8c40004eb0 con 0x7f8c5c070880 2024-03-25T19:47:17.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c5c13cea0 con 0x7f8c5c070880 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8c4c001ba0 con 0x7f8c5c070880 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:17.230 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:17.231 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:17.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:17.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 10 client.?.objecter ms_handle_connect 0x7f8c5c149a40 2024-03-25T19:47:17.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.230+0000 7f8c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3033992270 0 0) 0x7f8c5c13cea0 con 0x7f8c5c070880 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 >> v1:172.21.15.193:6789/0 conn(0x7f8c5c149a40 legacy=0x7f8c5c132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 >> v1:172.21.15.71:6789/0 conn(0x7f8c5c0704a0 legacy=0x7f8c5c1d5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 1 monclient: found mon.c 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c139190 con 0x7f8c5c070880 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 5 monclient: authenticate success, global_id 14528 2024-03-25T19:47:17.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 10 monclient: _renew_subs 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c1393e0 con 0x7f8c5c070880 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 10 client.14528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 10 monclient: _renew_subs 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c139e30 con 0x7f8c5c070880 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.245+0000 7f8c64563640 1 librados: init done 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.246+0000 7f8c64563640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:17.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.246+0000 7f8c64563640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:17.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.246+0000 7f8c64563640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:17.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.246+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5c0627b0 con 0x7f8c5c070880 2024-03-25T19:47:17.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.562+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c4c004a90 con 0x7f8c5c070880 2024-03-25T19:47:17.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.562+0000 7f8c52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:17.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.562+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8c4c004e20 con 0x7f8c5c070880 2024-03-25T19:47:17.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.562+0000 7f8c52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:17.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.562+0000 7f8c52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:17.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.562+0000 7f8c52ffd640 10 monclient: dump: 2024-03-25T19:47:17.562 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:17.562 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:17.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.562+0000 7f8c60afc640 4 set_mon_vals no callback set 2024-03-25T19:47:17.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.563+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f8c4c005040 con 0x7f8c5c070880 2024-03-25T19:47:17.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.563+0000 7f8c622ff640 10 monclient: get_auth_request con 0x7f8c4006dd00 auth_method 0 2024-03-25T19:47:17.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.563+0000 7f8c52ffd640 10 client.14528.objecter ms_handle_connect 0x7f8c4006dd00 2024-03-25T19:47:17.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.569+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f8c4c054a10 con 0x7f8c5c070880 2024-03-25T19:47:17.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.569+0000 7f8c52ffd640 10 client.14528.objecter ms_dispatch 0x7f8c5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:17.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.569+0000 7f8c52ffd640 3 client.14528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:17.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.569+0000 7f8c52ffd640 3 client.14528.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:17.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.569+0000 7f8c52ffd640 20 client.14528.objecter dump_active .. 0 homeless 2024-03-25T19:47:17.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.569+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c4c050520 con 0x7f8c5c070880 2024-03-25T19:47:17.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.569+0000 7f8c52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:17.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.569+0000 7f8c52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:17.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.838+0000 7f8c64563640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:17.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.838+0000 7f8c64563640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:17.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.838+0000 7f8c64563640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:17.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.838+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c5c13a3e0 con 0x7f8c5c070880 2024-03-25T19:47:17.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.875+0000 7f8c52ffd640 1 -- 172.21.15.71:0/1029608915 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f8c5c13a3e0 con 0x7f8c5c070880 2024-03-25T19:47:17.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.875+0000 7f8c52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:17.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.875+0000 7f8c52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:17.876 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:47:17.876 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:17.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.877+0000 7f8c64563640 10 librados: watch_flush enter 2024-03-25T19:47:17.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.877+0000 7f8c64563640 10 librados: watch_flush exit 2024-03-25T19:47:17.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.877+0000 7f8c64563640 20 client.14528.objecter shutdown clearing up homeless session... 2024-03-25T19:47:17.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.877+0000 7f8c64563640 10 client.14528.objecter successfully canceled tick 2024-03-25T19:47:17.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.877+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 >> v1:172.21.15.71:6800/3911735624 conn(0x7f8c4006dd00 legacy=0x7f8c400701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:17.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.877+0000 7f8c64563640 10 monclient: shutdown 2024-03-25T19:47:17.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.877+0000 7f8c64563640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:17.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.877+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 >> v1:172.21.15.71:6790/0 conn(0x7f8c5c070880 legacy=0x7f8c5c131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:17.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.878+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 shutdown_connections 2024-03-25T19:47:17.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.878+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 >> 172.21.15.71:0/1029608915 conn(0x7f8c5c0b5c00 msgr2=0x7f8c5c0b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:17.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.878+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 shutdown_connections 2024-03-25T19:47:17.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.878+0000 7f8c64563640 1 -- 172.21.15.71:0/1029608915 wait complete. 2024-03-25T19:47:17.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:17.878+0000 7f8c64563640 1 librados: shutdown 2024-03-25T19:47:17.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:22.446 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:05.202 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:22.495 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:05.251 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:22.529 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:05.285 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:22.548 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:05.304 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:22.556 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:05.275 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:22.557 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:05.313 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:22.560 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:05.279 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:22.584 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:05.341 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:22.594 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:05.313 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:24.696 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:24.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.794+0000 7fa995db6640 10 monclient: get_monmap_and_config 2024-03-25T19:47:24.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.794+0000 7fa995db6640 10 monclient: build_initial_monmap 2024-03-25T19:47:24.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.794+0000 7fa995db6640 10 monclient: monmap: 2024-03-25T19:47:24.794 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:47:24.795660+0000 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:47:24.795660+0000 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:47:24.795 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:24.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.795+0000 7fa995db6640 1 Processor -- start 2024-03-25T19:47:24.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.795+0000 7fa995db6640 1 -- start start 2024-03-25T19:47:24.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.795+0000 7fa995db6640 10 monclient: init 2024-03-25T19:47:24.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9900a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 10 monclient(hunting): picked mon.noname-a con 0x7fa99013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 10 monclient(hunting): picked mon.noname-b con 0x7fa9901430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa990142520 con 0x7fa99013f710 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9900b28a0 con 0x7fa9900a98f0 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9900b2a30 con 0x7fa9901430c0 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:24.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.796+0000 7fa995db6640 10 monclient(hunting): authenticate will time out at 5251.108196s 2024-03-25T19:47:24.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.810+0000 7fa98effd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fa99013f710 0x7fa9901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52834/0 (socket says 172.21.15.71:52834) 2024-03-25T19:47:24.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.810+0000 7fa98effd640 1 -- 172.21.15.71:0/1132463017 learned_addr learned my addr 172.21.15.71:0/1132463017 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:24.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.819+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837934857 0 0) 0x7fa9900b28a0 con 0x7fa9900a98f0 2024-03-25T19:47:24.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.819+0000 7fa98e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:24.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.819+0000 7fa98e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:24.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.819+0000 7fa98e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:24.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.819+0000 7fa98e7fc640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:47:24.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.819+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa978003520 con 0x7fa9900a98f0 2024-03-25T19:47:24.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3438824926 0 0) 0x7fa978003520 con 0x7fa9900a98f0 2024-03-25T19:47:24.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 >> v1:172.21.15.71:6790/0 conn(0x7fa9901430c0 legacy=0x7fa99014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:47:24.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 >> v1:172.21.15.71:6789/0 conn(0x7fa99013f710 legacy=0x7fa9901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9900b44c0 con 0x7fa9900a98f0 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa980001690 con 0x7fa9900a98f0 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.827+0000 7fa98e7fc640 10 monclient: dump: 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:24.828 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.828+0000 7fa995db6640 5 monclient: authenticate success, global_id 4525 2024-03-25T19:47:24.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.828+0000 7fa995db6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.843+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa980001860 con 0x7fa9900a98f0 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.843+0000 7fa98e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.843+0000 7fa98e7fc640 1 -- 172.21.15.71:0/1132463017 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa980001cb0 con 0x7fa9900a98f0 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.843+0000 7fa98e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.843+0000 7fa98e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.843+0000 7fa98e7fc640 10 monclient: dump: 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:24.843 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:24.844 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:24.844 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:24.844 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:24.844 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:24.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.843+0000 7fa995db6640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:24.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.843+0000 7fa995db6640 4 set_mon_vals no callback set 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.844+0000 7fa995db6640 10 monclient: shutdown 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.844+0000 7fa995db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.844+0000 7fa995db6640 1 -- 172.21.15.71:0/1132463017 >> v1:172.21.15.193:6789/0 conn(0x7fa9900a98f0 legacy=0x7fa9900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.844+0000 7fa995db6640 1 -- 172.21.15.71:0/1132463017 shutdown_connections 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.844+0000 7fa995db6640 1 -- 172.21.15.71:0/1132463017 >> 172.21.15.71:0/1132463017 conn(0x7fa9900b7ea0 msgr2=0x7fa9900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.844+0000 7fa995db6640 1 -- 172.21.15.71:0/1132463017 shutdown_connections 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.844+0000 7fa995db6640 1 -- 172.21.15.71:0/1132463017 wait complete. 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.845+0000 7fa995db6640 10 monclient: build_initial_monmap 2024-03-25T19:47:24.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.845+0000 7fa995db6640 10 monclient: monmap: 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:47:24.846 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.846+0000 7fa995db6640 1 librados: starting msgr at 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.846+0000 7fa995db6640 1 librados: starting objecter 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.846+0000 7fa995db6640 1 Processor -- start 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.846+0000 7fa995db6640 1 -- start start 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.846+0000 7fa995db6640 1 librados: setting wanted keys 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.846+0000 7fa995db6640 1 librados: calling monclient init 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.846+0000 7fa995db6640 10 monclient: init 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 10 monclient(hunting): picked mon.noname-a con 0x7fa9900a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 10 monclient(hunting): picked mon.noname-b con 0x7fa99013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9901430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9901399f0 con 0x7fa9900a98f0 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9901386a0 con 0x7fa99013f710 2024-03-25T19:47:24.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9900b1760 con 0x7fa9901430c0 2024-03-25T19:47:24.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:24.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.847+0000 7fa995db6640 10 monclient(hunting): authenticate will time out at 5251.159381s 2024-03-25T19:47:24.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.852+0000 7fa98ffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa9901430c0 0x7fa9901e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46820/0 (socket says 172.21.15.71:46820) 2024-03-25T19:47:24.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.852+0000 7fa98ffff640 1 -- 172.21.15.71:0/549261648 learned_addr learned my addr 172.21.15.71:0/549261648 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:24.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.853+0000 7fa977fff640 10 client.?.objecter ms_handle_connect 0x7fa9901430c0 2024-03-25T19:47:24.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.853+0000 7fa977fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:24.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.854+0000 7fa977fff640 10 client.?.objecter ms_handle_connect 0x7fa99013f710 2024-03-25T19:47:24.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.854+0000 7fa977fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.857+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660517786 0 0) 0x7fa9900b1760 con 0x7fa9901430c0 2024-03-25T19:47:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.857+0000 7fa977fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.857+0000 7fa977fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.857+0000 7fa977fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.857+0000 7fa977fff640 10 monclient(hunting): my global_id is 14531 2024-03-25T19:47:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.857+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa97c004eb0 con 0x7fa9901430c0 2024-03-25T19:47:24.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.862+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633522715 0 0) 0x7fa9901386a0 con 0x7fa99013f710 2024-03-25T19:47:24.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.862+0000 7fa977fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:24.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.862+0000 7fa977fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:24.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.862+0000 7fa977fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:24.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.862+0000 7fa977fff640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:47:24.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.862+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9900b1760 con 0x7fa99013f710 2024-03-25T19:47:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.867+0000 7fa977fff640 10 client.?.objecter ms_handle_connect 0x7fa9900a98f0 2024-03-25T19:47:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.867+0000 7fa977fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:24.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.871+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4043959875 0 0) 0x7fa97c004eb0 con 0x7fa9901430c0 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.871+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9901386a0 con 0x7fa9901430c0 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.871+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa988001ad0 con 0x7fa9901430c0 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.871+0000 7fa977fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.871+0000 7fa977fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.871+0000 7fa977fff640 10 monclient(hunting): dump: 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:24.871 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:24.872 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:24.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.871+0000 7fa977fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:24.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.873+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2792345747 0 0) 0x7fa9900b1760 con 0x7fa99013f710 2024-03-25T19:47:24.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.873+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa97c004eb0 con 0x7fa99013f710 2024-03-25T19:47:24.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.873+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa984001930 con 0x7fa99013f710 2024-03-25T19:47:24.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.873+0000 7fa977fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.873+0000 7fa977fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.873+0000 7fa977fff640 10 monclient(hunting): dump: 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:24.873 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:24.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1960173135 0 0) 0x7fa9901386a0 con 0x7fa9901430c0 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 >> v1:172.21.15.193:6789/0 conn(0x7fa99013f710 legacy=0x7fa9901da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 >> v1:172.21.15.71:6789/0 conn(0x7fa9900a98f0 legacy=0x7fa990137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 1 monclient: found mon.c 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9901380f0 con 0x7fa9901430c0 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa977fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 5 monclient: authenticate success, global_id 14531 2024-03-25T19:47:24.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 10 monclient: _renew_subs 2024-03-25T19:47:24.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:24.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9901383a0 con 0x7fa9901430c0 2024-03-25T19:47:24.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 10 client.14531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:24.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 10 monclient: _renew_subs 2024-03-25T19:47:24.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:24.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9901db170 con 0x7fa9901430c0 2024-03-25T19:47:24.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.877+0000 7fa995db6640 1 librados: init done 2024-03-25T19:47:24.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.879+0000 7fa995db6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:24.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.879+0000 7fa995db6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:24.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.879+0000 7fa995db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:24.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:24.879+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9901db470 con 0x7fa9901430c0 2024-03-25T19:47:25.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.192+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa988001fe0 con 0x7fa9901430c0 2024-03-25T19:47:25.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.192+0000 7fa977fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:25.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.192+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fa988004d80 con 0x7fa9901430c0 2024-03-25T19:47:25.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.192+0000 7fa977fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:25.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.192+0000 7fa977fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.192+0000 7fa977fff640 10 monclient: dump: 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:25.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.192+0000 7fa98dffb640 4 set_mon_vals no callback set 2024-03-25T19:47:25.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.193+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fa98801ad50 con 0x7fa9901430c0 2024-03-25T19:47:25.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.194+0000 7fa98f7fe640 10 monclient: get_auth_request con 0x7fa97c06df60 auth_method 0 2024-03-25T19:47:25.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.194+0000 7fa977fff640 10 client.14531.objecter ms_handle_connect 0x7fa97c06df60 2024-03-25T19:47:25.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.200+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fa988054a60 con 0x7fa9901430c0 2024-03-25T19:47:25.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.200+0000 7fa977fff640 10 client.14531.objecter ms_dispatch 0x7fa990000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:25.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.200+0000 7fa977fff640 3 client.14531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:25.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.200+0000 7fa977fff640 3 client.14531.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:25.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.200+0000 7fa977fff640 20 client.14531.objecter dump_active .. 0 homeless 2024-03-25T19:47:25.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.200+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9880503a0 con 0x7fa9901430c0 2024-03-25T19:47:25.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.200+0000 7fa977fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:25.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.200+0000 7fa977fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:25.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.464+0000 7fa995db6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:25.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.464+0000 7fa995db6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:25.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.464+0000 7fa995db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:25.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.464+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa99014f320 con 0x7fa9901430c0 2024-03-25T19:47:25.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.505+0000 7fa977fff640 1 -- 172.21.15.71:0/549261648 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fa9880524a0 con 0x7fa9901430c0 2024-03-25T19:47:25.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.505+0000 7fa977fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:25.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.505+0000 7fa977fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:25.506 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:47:25.506 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:25.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 10 librados: watch_flush enter 2024-03-25T19:47:25.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 10 librados: watch_flush exit 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 20 client.14531.objecter shutdown clearing up homeless session... 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 10 client.14531.objecter successfully canceled tick 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 >> v1:172.21.15.71:6800/3911735624 conn(0x7fa97c06df60 legacy=0x7fa97c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 10 monclient: shutdown 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 >> v1:172.21.15.71:6790/0 conn(0x7fa9901430c0 legacy=0x7fa9901e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 shutdown_connections 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 >> 172.21.15.71:0/549261648 conn(0x7fa9900b7ea0 msgr2=0x7fa990074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 shutdown_connections 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 1 -- 172.21.15.71:0/549261648 wait complete. 2024-03-25T19:47:25.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:25.508+0000 7fa995db6640 1 librados: shutdown 2024-03-25T19:47:25.521 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:32.328 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:32.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.443+0000 7f25f543e640 10 monclient: get_monmap_and_config 2024-03-25T19:47:32.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.443+0000 7f25f543e640 10 monclient: build_initial_monmap 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.443+0000 7f25f543e640 10 monclient: monmap: 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:47:32.444795+0000 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:47:32.444795+0000 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:47:32.444 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:32.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.444+0000 7f25f543e640 1 Processor -- start 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.444+0000 7f25f543e640 1 -- start start 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.444+0000 7f25f543e640 10 monclient: init 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 10 monclient(hunting): picked mon.noname-b con 0x7f25f006c380 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 10 monclient(hunting): picked mon.noname-c con 0x7f25f00b12d0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 10 monclient(hunting): picked mon.noname-a con 0x7f25f00aedf0 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f00b16f0 con 0x7f25f00aedf0 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f00adfe0 con 0x7f25f00b12d0 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f00add00 con 0x7f25f006c380 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:32.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.445+0000 7f25f543e640 10 monclient(hunting): authenticate will time out at 5258.757755s 2024-03-25T19:47:32.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.464+0000 7f25e7fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f25f00b12d0 0x7f25f00ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:52378/0 (socket says 172.21.15.71:52378) 2024-03-25T19:47:32.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.464+0000 7f25e7fff640 1 -- 172.21.15.71:0/1632720666 learned_addr learned my addr 172.21.15.71:0/1632720666 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:32.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.473+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190216954 0 0) 0x7f25f00add00 con 0x7f25f006c380 2024-03-25T19:47:32.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.473+0000 7f25ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:32.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.473+0000 7f25ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:32.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.473+0000 7f25ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:32.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.473+0000 7f25ee7fc640 10 monclient(hunting): my global_id is 14534 2024-03-25T19:47:32.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.473+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc003520 con 0x7f25f006c380 2024-03-25T19:47:32.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.482+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417171237 0 0) 0x7f25f00adfe0 con 0x7f25f00b12d0 2024-03-25T19:47:32.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.482+0000 7f25ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:32.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.482+0000 7f25ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:32.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.482+0000 7f25ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:32.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.482+0000 7f25ee7fc640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:47:32.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.482+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f00add00 con 0x7f25f00b12d0 2024-03-25T19:47:32.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340607513 0 0) 0x7f25dc003520 con 0x7f25f006c380 2024-03-25T19:47:32.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 >> v1:172.21.15.193:6789/0 conn(0x7f25f00b12d0 legacy=0x7f25f00ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:32.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 >> v1:172.21.15.71:6789/0 conn(0x7f25f00aedf0 legacy=0x7f25f014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 1 monclient: found mon.noname-b 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f00ae2c0 con 0x7f25f006c380 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f25e0001690 con 0x7f25f006c380 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25ee7fc640 10 monclient: dump: 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:32.484 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25f543e640 5 monclient: authenticate success, global_id 14534 2024-03-25T19:47:32.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.483+0000 7f25f543e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.488+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25e0001870 con 0x7f25f006c380 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.488+0000 7f25ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.488+0000 7f25ee7fc640 1 -- 172.21.15.71:0/1632720666 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f25e0001c00 con 0x7f25f006c380 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.488+0000 7f25f543e640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.488+0000 7f25f543e640 4 set_mon_vals no callback set 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.488+0000 7f25ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.488+0000 7f25ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.488+0000 7f25ee7fc640 10 monclient: dump: 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:32.488 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:32.489 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:32.489 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:32.489 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:32.489 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:32.489 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:32.489 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:32.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.489+0000 7f25f543e640 10 monclient: shutdown 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.489+0000 7f25f543e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.489+0000 7f25f543e640 1 -- 172.21.15.71:0/1632720666 >> v1:172.21.15.71:6790/0 conn(0x7f25f006c380 legacy=0x7f25f00b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.489+0000 7f25f543e640 1 -- 172.21.15.71:0/1632720666 shutdown_connections 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.489+0000 7f25f543e640 1 -- 172.21.15.71:0/1632720666 >> 172.21.15.71:0/1632720666 conn(0x7f25f00b5d10 msgr2=0x7f25f00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.489+0000 7f25f543e640 1 -- 172.21.15.71:0/1632720666 shutdown_connections 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.489+0000 7f25f543e640 1 -- 172.21.15.71:0/1632720666 wait complete. 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 10 monclient: build_initial_monmap 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 10 monclient: monmap: 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:32.492 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 1 librados: starting msgr at 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 1 librados: starting objecter 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 1 Processor -- start 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 1 -- start start 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 1 librados: setting wanted keys 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 1 librados: calling monclient init 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.490+0000 7f25f543e640 10 monclient: init 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:32.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:32.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 10 monclient(hunting): picked mon.noname-c con 0x7f25f006c380 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:32.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 10 monclient(hunting): picked mon.noname-b con 0x7f25f00aedf0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:32.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 10 monclient(hunting): picked mon.noname-a con 0x7f25f00b12d0 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:32.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f006c8e0 con 0x7f25f00b12d0 2024-03-25T19:47:32.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f01a3cc0 con 0x7f25f00aedf0 2024-03-25T19:47:32.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f01a2720 con 0x7f25f006c380 2024-03-25T19:47:32.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:32.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.491+0000 7f25f543e640 10 monclient(hunting): authenticate will time out at 5258.803319s 2024-03-25T19:47:32.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.511+0000 7f25ef7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f25f00b12d0 0x7f25f01e2de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:38076/0 (socket says 172.21.15.71:38076) 2024-03-25T19:47:32.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.511+0000 7f25ef7fe640 1 -- 172.21.15.71:0/3432069664 learned_addr learned my addr 172.21.15.71:0/3432069664 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:32.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.515+0000 7f25e77fe640 10 client.?.objecter ms_handle_connect 0x7f25f00b12d0 2024-03-25T19:47:32.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.515+0000 7f25e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:32.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.519+0000 7f25e77fe640 10 client.?.objecter ms_handle_connect 0x7f25f00aedf0 2024-03-25T19:47:32.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.519+0000 7f25e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:32.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.524+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57169810 0 0) 0x7f25f01a3cc0 con 0x7f25f00aedf0 2024-03-25T19:47:32.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.524+0000 7f25e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:32.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.524+0000 7f25e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:32.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.524+0000 7f25e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:32.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.524+0000 7f25e77fe640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:47:32.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.524+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d0004eb0 con 0x7f25f00aedf0 2024-03-25T19:47:32.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.533+0000 7f25e77fe640 10 client.?.objecter ms_handle_connect 0x7f25f006c380 2024-03-25T19:47:32.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.533+0000 7f25e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.539+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654422798 0 0) 0x7f25d0004eb0 con 0x7f25f00aedf0 2024-03-25T19:47:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25f01a3cc0 con 0x7f25f00aedf0 2024-03-25T19:47:32.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f25d8001890 con 0x7f25f00aedf0 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 10 monclient(hunting): dump: 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:32.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:32.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862073002 0 0) 0x7f25f006c8e0 con 0x7f25f00b12d0 2024-03-25T19:47:32.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:32.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:32.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:32.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:47:32.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.540+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d0004eb0 con 0x7f25f00b12d0 2024-03-25T19:47:32.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2319471563 0 0) 0x7f25f01a3cc0 con 0x7f25f00aedf0 2024-03-25T19:47:32.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 >> v1:172.21.15.71:6790/0 conn(0x7f25f006c380 legacy=0x7f25f01daa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:32.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 >> v1:172.21.15.71:6789/0 conn(0x7f25f00b12d0 legacy=0x7f25f01e2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:32.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:32.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 1 monclient: found mon.b 2024-03-25T19:47:32.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:32.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:32.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f01c5120 con 0x7f25f00aedf0 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 5 monclient: authenticate success, global_id 4534 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 10 monclient: _renew_subs 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f01db170 con 0x7f25f00aedf0 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 10 monclient: _renew_subs 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f01a4330 con 0x7f25f00aedf0 2024-03-25T19:47:32.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.550+0000 7f25f543e640 1 librados: init done 2024-03-25T19:47:32.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.551+0000 7f25f543e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:32.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.551+0000 7f25f543e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:32.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.551+0000 7f25f543e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:32.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.551+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f006d790 con 0x7f25f00aedf0 2024-03-25T19:47:32.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.835+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d8001d40 con 0x7f25f00aedf0 2024-03-25T19:47:32.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.835+0000 7f25e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:32.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.835+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f25d80048a0 con 0x7f25f00aedf0 2024-03-25T19:47:32.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.835+0000 7f25e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:32.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.835+0000 7f25e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.835+0000 7f25e77fe640 10 monclient: dump: 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:32.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.835+0000 7f25edffb640 4 set_mon_vals no callback set 2024-03-25T19:47:32.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.836+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f25d801a930 con 0x7f25f00aedf0 2024-03-25T19:47:32.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.837+0000 7f25eeffd640 10 monclient: get_auth_request con 0x7f25d006de20 auth_method 0 2024-03-25T19:47:32.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.837+0000 7f25e77fe640 10 client.4534.objecter ms_handle_connect 0x7f25d006de20 2024-03-25T19:47:32.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.842+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f25d8054570 con 0x7f25f00aedf0 2024-03-25T19:47:32.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.842+0000 7f25e77fe640 10 client.4534.objecter ms_dispatch 0x7f25f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:32.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.842+0000 7f25e77fe640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:32.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.842+0000 7f25e77fe640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:32.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.842+0000 7f25e77fe640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T19:47:32.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.843+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25d804feb0 con 0x7f25f00aedf0 2024-03-25T19:47:32.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.843+0000 7f25e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:32.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:32.843+0000 7f25e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:33.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.100+0000 7f25f543e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:33.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.100+0000 7f25f543e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:33.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.100+0000 7f25f543e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:33.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.100+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25f01a3e70 con 0x7f25f00aedf0 2024-03-25T19:47:33.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.121+0000 7f25e77fe640 1 -- 172.21.15.71:0/3432069664 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f25f01a3e70 con 0x7f25f00aedf0 2024-03-25T19:47:33.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.121+0000 7f25e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:33.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.121+0000 7f25e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:33.121 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:47:33.121 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:33.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.123+0000 7f25f543e640 10 librados: watch_flush enter 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.123+0000 7f25f543e640 10 librados: watch_flush exit 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.123+0000 7f25f543e640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.123+0000 7f25f543e640 10 client.4534.objecter successfully canceled tick 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.123+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 >> v1:172.21.15.71:6800/3911735624 conn(0x7f25d006de20 legacy=0x7f25d00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.123+0000 7f25f543e640 10 monclient: shutdown 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.123+0000 7f25f543e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.123+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 >> v1:172.21.15.193:6789/0 conn(0x7f25f00aedf0 legacy=0x7f25f01c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.124+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 shutdown_connections 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.124+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 >> 172.21.15.71:0/3432069664 conn(0x7f25f00b5d10 msgr2=0x7f25f00ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.124+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 shutdown_connections 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.124+0000 7f25f543e640 1 -- 172.21.15.71:0/3432069664 wait complete. 2024-03-25T19:47:33.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:33.124+0000 7f25f543e640 1 librados: shutdown 2024-03-25T19:47:33.137 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:39.944 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:40.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.058+0000 7fd340938640 10 monclient: get_monmap_and_config 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.058+0000 7fd340938640 10 monclient: build_initial_monmap 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.058+0000 7fd340938640 10 monclient: monmap: 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:47:40.059969+0000 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:47:40.059969+0000 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:40.059 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:40.060 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:47:40.060 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:47:40.060 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.058+0000 7fd340938640 1 Processor -- start 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 1 -- start start 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 10 monclient: init 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 10 monclient(hunting): picked mon.noname-b con 0x7fd33c13f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 10 monclient(hunting): picked mon.noname-a con 0x7fd33c1430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c142520 con 0x7fd33c1430c0 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0b28a0 con 0x7fd33c0a98f0 2024-03-25T19:47:40.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0b2a30 con 0x7fd33c13f710 2024-03-25T19:47:40.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:40.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.059+0000 7fd340938640 10 monclient(hunting): authenticate will time out at 5266.372633s 2024-03-25T19:47:40.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.062+0000 7fd33a59c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd33c0a98f0 0x7fd33c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:55798/0 (socket says 172.21.15.71:55798) 2024-03-25T19:47:40.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.062+0000 7fd33a59c640 1 -- 172.21.15.71:0/1686079253 learned_addr learned my addr 172.21.15.71:0/1686079253 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:40.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.071+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146627099 0 0) 0x7fd33c0b28a0 con 0x7fd33c0a98f0 2024-03-25T19:47:40.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.071+0000 7fd339d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:47:40.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.071+0000 7fd339d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:40.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.071+0000 7fd339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:40.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.071+0000 7fd339d9b640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:47:40.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.071+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd328003520 con 0x7fd33c0a98f0 2024-03-25T19:47:40.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154961770 0 0) 0x7fd33c142520 con 0x7fd33c1430c0 2024-03-25T19:47:40.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:47:40.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:40.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:40.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:47:40.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c0b28a0 con 0x7fd33c1430c0 2024-03-25T19:47:40.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091505185 0 0) 0x7fd33c0b2a30 con 0x7fd33c13f710 2024-03-25T19:47:40.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:47:40.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:40.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:40.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 10 monclient(hunting): my global_id is 14540 2024-03-25T19:47:40.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.076+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c142520 con 0x7fd33c13f710 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4089349934 0 0) 0x7fd328003520 con 0x7fd33c0a98f0 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 >> v1:172.21.15.71:6790/0 conn(0x7fd33c13f710 legacy=0x7fd33c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 >> v1:172.21.15.71:6789/0 conn(0x7fd33c1430c0 legacy=0x7fd33c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 1 monclient: found mon.noname-c 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0b44c0 con 0x7fd33c0a98f0 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 10 monclient: _finish_auth 0 2024-03-25T19:47:40.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd32c001690 con 0x7fd33c0a98f0 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd339d9b640 10 monclient: dump: 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:40.080 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:40.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd340938640 5 monclient: authenticate success, global_id 4537 2024-03-25T19:47:40.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.078+0000 7fd340938640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:40.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.089+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd32c0017f0 con 0x7fd33c0a98f0 2024-03-25T19:47:40.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.089+0000 7fd339d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:40.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.089+0000 7fd339d9b640 1 -- 172.21.15.71:0/1686079253 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd32c001c00 con 0x7fd33c0a98f0 2024-03-25T19:47:40.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.089+0000 7fd339d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:40.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.089+0000 7fd339d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:40.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.089+0000 7fd339d9b640 10 monclient: dump: 2024-03-25T19:47:40.090 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:40.090 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.089+0000 7fd340938640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:40.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.089+0000 7fd340938640 4 set_mon_vals no callback set 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.090+0000 7fd340938640 10 monclient: shutdown 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.090+0000 7fd340938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.090+0000 7fd340938640 1 -- 172.21.15.71:0/1686079253 >> v1:172.21.15.193:6789/0 conn(0x7fd33c0a98f0 legacy=0x7fd33c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.090+0000 7fd340938640 1 -- 172.21.15.71:0/1686079253 shutdown_connections 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.090+0000 7fd340938640 1 -- 172.21.15.71:0/1686079253 >> 172.21.15.71:0/1686079253 conn(0x7fd33c0b7ea0 msgr2=0x7fd33c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.090+0000 7fd340938640 1 -- 172.21.15.71:0/1686079253 shutdown_connections 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.090+0000 7fd340938640 1 -- 172.21.15.71:0/1686079253 wait complete. 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.091+0000 7fd340938640 10 monclient: build_initial_monmap 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.091+0000 7fd340938640 10 monclient: monmap: 2024-03-25T19:47:40.093 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 1 librados: starting msgr at 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 1 librados: starting objecter 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 1 Processor -- start 2024-03-25T19:47:40.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 1 -- start start 2024-03-25T19:47:40.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 1 librados: setting wanted keys 2024-03-25T19:47:40.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 1 librados: calling monclient init 2024-03-25T19:47:40.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 10 monclient: init 2024-03-25T19:47:40.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.092+0000 7fd340938640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.093+0000 7fd340938640 10 monclient(hunting): picked mon.noname-b con 0x7fd33c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.093+0000 7fd340938640 10 monclient(hunting): picked mon.noname-a con 0x7fd33c13f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.093+0000 7fd340938640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c1430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.093+0000 7fd340938640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c135730 con 0x7fd33c13f710 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.093+0000 7fd340938640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c1358c0 con 0x7fd33c0a98f0 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.093+0000 7fd340938640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c135a50 con 0x7fd33c1430c0 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.093+0000 7fd340938640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:40.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.093+0000 7fd340938640 10 monclient(hunting): authenticate will time out at 5266.405925s 2024-03-25T19:47:40.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.099+0000 7fd33ad9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd33c1430c0 0x7fd33c1d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49682/0 (socket says 172.21.15.71:49682) 2024-03-25T19:47:40.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.099+0000 7fd33ad9d640 1 -- 172.21.15.71:0/3025757320 learned_addr learned my addr 172.21.15.71:0/3025757320 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:40.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.105+0000 7fd332ffd640 10 client.?.objecter ms_handle_connect 0x7fd33c1430c0 2024-03-25T19:47:40.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.105+0000 7fd332ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:40.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.107+0000 7fd332ffd640 10 client.?.objecter ms_handle_connect 0x7fd33c13f710 2024-03-25T19:47:40.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.107+0000 7fd332ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:40.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.112+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971979317 0 0) 0x7fd33c135a50 con 0x7fd33c1430c0 2024-03-25T19:47:40.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.112+0000 7fd332ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:40.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.112+0000 7fd332ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:40.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.112+0000 7fd332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:40.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.112+0000 7fd332ffd640 10 monclient(hunting): my global_id is 14543 2024-03-25T19:47:40.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.112+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd31c004eb0 con 0x7fd33c1430c0 2024-03-25T19:47:40.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.116+0000 7fd332ffd640 10 client.?.objecter ms_handle_connect 0x7fd33c0a98f0 2024-03-25T19:47:40.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.116+0000 7fd332ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.126+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130330019 0 0) 0x7fd31c004eb0 con 0x7fd33c1430c0 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.126+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd33c135a50 con 0x7fd33c1430c0 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.126+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd334001c90 con 0x7fd33c1430c0 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.126+0000 7fd332ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.126+0000 7fd332ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.126+0000 7fd332ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:40.127 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:40.128 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:40.128 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:40.128 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:40.128 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:40.128 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:40.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.126+0000 7fd332ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:40.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.128+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017514206 0 0) 0x7fd33c135730 con 0x7fd33c13f710 2024-03-25T19:47:40.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.128+0000 7fd332ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:40.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.128+0000 7fd332ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:40.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.128+0000 7fd332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:40.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.128+0000 7fd332ffd640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:47:40.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.128+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd31c004eb0 con 0x7fd33c13f710 2024-03-25T19:47:40.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.135+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432847361 0 0) 0x7fd33c1358c0 con 0x7fd33c0a98f0 2024-03-25T19:47:40.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.135+0000 7fd332ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:40.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.135+0000 7fd332ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:40.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.135+0000 7fd332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:40.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.135+0000 7fd332ffd640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:47:40.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.135+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c135730 con 0x7fd33c0a98f0 2024-03-25T19:47:40.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1346184686 0 0) 0x7fd33c135a50 con 0x7fd33c1430c0 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 >> v1:172.21.15.193:6789/0 conn(0x7fd33c0a98f0 legacy=0x7fd33c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 >> v1:172.21.15.71:6789/0 conn(0x7fd33c13f710 legacy=0x7fd33c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 1 monclient: found mon.c 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c1d5000 con 0x7fd33c1430c0 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd332ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 5 monclient: authenticate success, global_id 14543 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 10 monclient: _renew_subs 2024-03-25T19:47:40.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c1d52b0 con 0x7fd33c1430c0 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 10 client.14543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 10 monclient: _renew_subs 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c13c900 con 0x7fd33c1430c0 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.142+0000 7fd340938640 1 librados: init done 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.143+0000 7fd340938640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.143+0000 7fd340938640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:40.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.143+0000 7fd340938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:40.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.143+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c13cbb0 con 0x7fd33c1430c0 2024-03-25T19:47:40.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.443+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd334003970 con 0x7fd33c1430c0 2024-03-25T19:47:40.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.443+0000 7fd332ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:40.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.443+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd334003ba0 con 0x7fd33c1430c0 2024-03-25T19:47:40.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.443+0000 7fd332ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:40.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.443+0000 7fd332ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.443+0000 7fd332ffd640 10 monclient: dump: 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.443+0000 7fd33959a640 4 set_mon_vals no callback set 2024-03-25T19:47:40.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.444+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fd334003dc0 con 0x7fd33c1430c0 2024-03-25T19:47:40.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.444+0000 7fd33a59c640 10 monclient: get_auth_request con 0x7fd31c06de80 auth_method 0 2024-03-25T19:47:40.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.444+0000 7fd332ffd640 10 client.14543.objecter ms_handle_connect 0x7fd31c06de80 2024-03-25T19:47:40.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.449+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fd334054980 con 0x7fd33c1430c0 2024-03-25T19:47:40.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.449+0000 7fd332ffd640 10 client.14543.objecter ms_dispatch 0x7fd33c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:40.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.449+0000 7fd332ffd640 3 client.14543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:40.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.449+0000 7fd332ffd640 3 client.14543.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:40.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.449+0000 7fd332ffd640 20 client.14543.objecter dump_active .. 0 homeless 2024-03-25T19:47:40.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.449+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd334050490 con 0x7fd33c1430c0 2024-03-25T19:47:40.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.449+0000 7fd332ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:40.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.449+0000 7fd332ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:40.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.703+0000 7fd340938640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:40.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.703+0000 7fd340938640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:40.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.703+0000 7fd340938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:47:40.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.703+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd33c1371c0 con 0x7fd33c1430c0 2024-03-25T19:47:40.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.732+0000 7fd332ffd640 1 -- 172.21.15.71:0/3025757320 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fd3340523c0 con 0x7fd33c1430c0 2024-03-25T19:47:40.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.732+0000 7fd332ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:40.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.732+0000 7fd332ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:40.733 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:47:40.733 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:40.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 10 librados: watch_flush enter 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 10 librados: watch_flush exit 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 20 client.14543.objecter shutdown clearing up homeless session... 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 10 client.14543.objecter successfully canceled tick 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 >> v1:172.21.15.71:6800/3911735624 conn(0x7fd31c06de80 legacy=0x7fd31c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 10 monclient: shutdown 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 >> v1:172.21.15.71:6790/0 conn(0x7fd33c1430c0 legacy=0x7fd33c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 shutdown_connections 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 >> 172.21.15.71:0/3025757320 conn(0x7fd33c0b7ea0 msgr2=0x7fd33c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 shutdown_connections 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 1 -- 172.21.15.71:0/3025757320 wait complete. 2024-03-25T19:47:40.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:40.735+0000 7fd340938640 1 librados: shutdown 2024-03-25T19:47:40.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:42.519 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:25.275 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.529 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:25.248 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.571 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:25.327 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.574 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:25.293 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.596 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:25.315 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.605 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:25.362 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.642 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:25.398 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.652 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:25.408 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.679 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:25.435 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:47.554 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.668+0000 7fdeda94b640 10 monclient: get_monmap_and_config 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.668+0000 7fdeda94b640 10 monclient: build_initial_monmap 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.668+0000 7fdeda94b640 10 monclient: monmap: 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:47:47.669196+0000 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:47:47.669196+0000 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:47.668 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:47.669 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:47:47.669 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:47:47.669 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:47.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.669+0000 7fdeda94b640 1 Processor -- start 2024-03-25T19:47:47.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.669+0000 7fdeda94b640 1 -- start start 2024-03-25T19:47:47.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.669+0000 7fdeda94b640 10 monclient: init 2024-03-25T19:47:47.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.669+0000 7fdeda94b640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:47.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.669+0000 7fdeda94b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:47.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.669+0000 7fdeda94b640 10 monclient(hunting): picked mon.noname-b con 0x7fded40a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:47.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.669+0000 7fdeda94b640 10 monclient(hunting): picked mon.noname-a con 0x7fded413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:47.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.670+0000 7fdeda94b640 10 monclient(hunting): picked mon.noname-c con 0x7fded41430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:47.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.670+0000 7fdeda94b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded4142520 con 0x7fded413f710 2024-03-25T19:47:47.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.670+0000 7fdeda94b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded40b28a0 con 0x7fded41430c0 2024-03-25T19:47:47.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.670+0000 7fdeda94b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded40b2a30 con 0x7fded40a98f0 2024-03-25T19:47:47.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.670+0000 7fdeda94b640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:47.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.670+0000 7fdeda94b640 10 monclient(hunting): authenticate will time out at 5273.982086s 2024-03-25T19:47:47.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.678+0000 7fded3fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fded40a98f0 0x7fded40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:43958/0 (socket says 172.21.15.71:43958) 2024-03-25T19:47:47.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.678+0000 7fded3fff640 1 -- 172.21.15.71:0/1937366548 learned_addr learned my addr 172.21.15.71:0/1937366548 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:47.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.694+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927601278 0 0) 0x7fded40b2a30 con 0x7fded40a98f0 2024-03-25T19:47:47.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.694+0000 7fded2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:47.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.694+0000 7fded2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:47.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.694+0000 7fded2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:47.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.694+0000 7fded2ffd640 10 monclient(hunting): my global_id is 14546 2024-03-25T19:47:47.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.694+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdebc003520 con 0x7fded40a98f0 2024-03-25T19:47:47.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.702+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030526440 0 0) 0x7fded4142520 con 0x7fded413f710 2024-03-25T19:47:47.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.702+0000 7fded2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:47.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.702+0000 7fded2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:47.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.702+0000 7fded2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:47.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.702+0000 7fded2ffd640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:47:47.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.702+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded40b2a30 con 0x7fded413f710 2024-03-25T19:47:47.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225370544 0 0) 0x7fdebc003520 con 0x7fded40a98f0 2024-03-25T19:47:47.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 >> v1:172.21.15.193:6789/0 conn(0x7fded41430c0 legacy=0x7fded414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:47.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 >> v1:172.21.15.71:6789/0 conn(0x7fded413f710 legacy=0x7fded41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:47.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded40b44c0 con 0x7fded40a98f0 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fdec4001690 con 0x7fded40a98f0 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fded2ffd640 10 monclient: dump: 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:47.705 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fdeda94b640 5 monclient: authenticate success, global_id 14546 2024-03-25T19:47:47.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.704+0000 7fdeda94b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:47.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdec40017f0 con 0x7fded40a98f0 2024-03-25T19:47:47.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fded2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:47.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fded2ffd640 1 -- 172.21.15.71:0/1937366548 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fdec4001c00 con 0x7fded40a98f0 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fded2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fded2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fded2ffd640 10 monclient: dump: 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:47.721 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:47.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fdeda94b640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:47.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fdeda94b640 4 set_mon_vals no callback set 2024-03-25T19:47:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fdeda94b640 10 monclient: shutdown 2024-03-25T19:47:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fdeda94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fdeda94b640 1 -- 172.21.15.71:0/1937366548 >> v1:172.21.15.71:6790/0 conn(0x7fded40a98f0 legacy=0x7fded40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fdeda94b640 1 -- 172.21.15.71:0/1937366548 shutdown_connections 2024-03-25T19:47:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.721+0000 7fdeda94b640 1 -- 172.21.15.71:0/1937366548 >> 172.21.15.71:0/1937366548 conn(0x7fded40b7ea0 msgr2=0x7fded40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:47.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.722+0000 7fdeda94b640 1 -- 172.21.15.71:0/1937366548 shutdown_connections 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.722+0000 7fdeda94b640 1 -- 172.21.15.71:0/1937366548 wait complete. 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.722+0000 7fdeda94b640 10 monclient: build_initial_monmap 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.722+0000 7fdeda94b640 10 monclient: monmap: 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:47.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.723+0000 7fdeda94b640 1 librados: starting msgr at 2024-03-25T19:47:47.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.723+0000 7fdeda94b640 1 librados: starting objecter 2024-03-25T19:47:47.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.723+0000 7fdeda94b640 1 Processor -- start 2024-03-25T19:47:47.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.723+0000 7fdeda94b640 1 -- start start 2024-03-25T19:47:47.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.723+0000 7fdeda94b640 1 librados: setting wanted keys 2024-03-25T19:47:47.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.723+0000 7fdeda94b640 1 librados: calling monclient init 2024-03-25T19:47:47.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.723+0000 7fdeda94b640 10 monclient: init 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 10 monclient(hunting): picked mon.noname-b con 0x7fded40a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 10 monclient(hunting): picked mon.noname-a con 0x7fded413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 10 monclient(hunting): picked mon.noname-c con 0x7fded41430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded4137ab0 con 0x7fded413f710 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded4138bd0 con 0x7fded40a98f0 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded4138e40 con 0x7fded41430c0 2024-03-25T19:47:47.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:47.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.724+0000 7fdeda94b640 10 monclient(hunting): authenticate will time out at 5274.036528s 2024-03-25T19:47:47.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.727+0000 7fded8ee8640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fded41430c0 0x7fded41e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:43994/0 (socket says 172.21.15.71:43994) 2024-03-25T19:47:47.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.727+0000 7fded8ee8640 1 -- 172.21.15.71:0/285496789 learned_addr learned my addr 172.21.15.71:0/285496789 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:47.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.728+0000 7fded0ff9640 10 client.?.objecter ms_handle_connect 0x7fded41430c0 2024-03-25T19:47:47.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.728+0000 7fded0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:47.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.741+0000 7fded0ff9640 10 client.?.objecter ms_handle_connect 0x7fded40a98f0 2024-03-25T19:47:47.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.741+0000 7fded0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:47.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.742+0000 7fded0ff9640 10 client.?.objecter ms_handle_connect 0x7fded413f710 2024-03-25T19:47:47.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.742+0000 7fded0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:47.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.750+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796270819 0 0) 0x7fded4138e40 con 0x7fded41430c0 2024-03-25T19:47:47.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.750+0000 7fded0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:47.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.750+0000 7fded0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:47.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.750+0000 7fded0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:47.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.750+0000 7fded0ff9640 10 monclient(hunting): my global_id is 14549 2024-03-25T19:47:47.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.750+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec0004eb0 con 0x7fded41430c0 2024-03-25T19:47:47.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.755+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820426736 0 0) 0x7fded4137ab0 con 0x7fded413f710 2024-03-25T19:47:47.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.755+0000 7fded0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:47.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.755+0000 7fded0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:47.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.755+0000 7fded0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:47.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.755+0000 7fded0ff9640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:47:47.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.755+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded4138e40 con 0x7fded413f710 2024-03-25T19:47:47.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.764+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052579120 0 0) 0x7fded4138bd0 con 0x7fded40a98f0 2024-03-25T19:47:47.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.764+0000 7fded0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:47.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.764+0000 7fded0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:47.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.764+0000 7fded0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:47.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.764+0000 7fded0ff9640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:47:47.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.764+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded4137ab0 con 0x7fded40a98f0 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.771+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477722981 0 0) 0x7fded4138e40 con 0x7fded413f710 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.771+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fded4138bd0 con 0x7fded413f710 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.771+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fdec80019f0 con 0x7fded413f710 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.771+0000 7fded0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.771+0000 7fded0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.771+0000 7fded0ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:47.771 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:47.772 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:47.772 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:47.772 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:47.772 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:47.772 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:47.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.771+0000 7fded0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.777+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1173873333 0 0) 0x7fdec0004eb0 con 0x7fded41430c0 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.777+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fded4138e40 con 0x7fded41430c0 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.777+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fdecc001c30 con 0x7fded41430c0 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.777+0000 7fded0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.777+0000 7fded0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.777+0000 7fded0ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:47.777 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:47.778 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:47.778 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:47.778 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:47.778 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:47.778 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:47.778 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.781+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2548765212 0 0) 0x7fded4138bd0 con 0x7fded413f710 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 >> v1:172.21.15.71:6790/0 conn(0x7fded41430c0 legacy=0x7fded41e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 >> v1:172.21.15.193:6789/0 conn(0x7fded40a98f0 legacy=0x7fded4137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 1 monclient: found mon.a 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded4139120 con 0x7fded413f710 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:47.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fded0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 5 monclient: authenticate success, global_id 4551 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 10 monclient: _renew_subs 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded4139400 con 0x7fded413f710 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 10 monclient: _renew_subs 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fded40b7880 con 0x7fded413f710 2024-03-25T19:47:47.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.782+0000 7fdeda94b640 1 librados: init done 2024-03-25T19:47:47.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.783+0000 7fdeda94b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:47.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.783+0000 7fdeda94b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:47.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.783+0000 7fdeda94b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:47:47.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:47.783+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded4137810 con 0x7fded413f710 2024-03-25T19:47:48.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.090+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdec8003610 con 0x7fded413f710 2024-03-25T19:47:48.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.090+0000 7fded0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:48.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.090+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fdec80039e0 con 0x7fded413f710 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.090+0000 7fded0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.090+0000 7fded0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.090+0000 7fded0ff9640 10 monclient: dump: 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:48.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.090+0000 7fded27fc640 4 set_mon_vals no callback set 2024-03-25T19:47:48.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.091+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fdec8003c00 con 0x7fded413f710 2024-03-25T19:47:48.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.092+0000 7fded3fff640 10 monclient: get_auth_request con 0x7fdec006df70 auth_method 0 2024-03-25T19:47:48.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.092+0000 7fded0ff9640 10 client.4551.objecter ms_handle_connect 0x7fdec006df70 2024-03-25T19:47:48.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.100+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fdec80535c0 con 0x7fded413f710 2024-03-25T19:47:48.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.100+0000 7fded0ff9640 10 client.4551.objecter ms_dispatch 0x7fded4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:48.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.100+0000 7fded0ff9640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:48.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.100+0000 7fded0ff9640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:48.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.100+0000 7fded0ff9640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T19:47:48.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.100+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdec804f0d0 con 0x7fded413f710 2024-03-25T19:47:48.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.100+0000 7fded0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:48.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.100+0000 7fded0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:48.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.330+0000 7fdeda94b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:48.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.330+0000 7fdeda94b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:48.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.330+0000 7fdeda94b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:47:48.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.330+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fded4137c60 con 0x7fded413f710 2024-03-25T19:47:48.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.363+0000 7fded0ff9640 1 -- 172.21.15.71:0/285496789 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fded4137c60 con 0x7fded413f710 2024-03-25T19:47:48.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.363+0000 7fded0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:48.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.363+0000 7fded0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:48.363 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:47:48.364 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:48.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 10 librados: watch_flush enter 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 10 librados: watch_flush exit 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 10 client.4551.objecter successfully canceled tick 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 >> v1:172.21.15.71:6800/3911735624 conn(0x7fdec006df70 legacy=0x7fdec0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 10 monclient: shutdown 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 >> v1:172.21.15.71:6789/0 conn(0x7fded413f710 legacy=0x7fded41df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 shutdown_connections 2024-03-25T19:47:48.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 >> 172.21.15.71:0/285496789 conn(0x7fded40b7ea0 msgr2=0x7fded4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:48.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 shutdown_connections 2024-03-25T19:47:48.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.366+0000 7fdeda94b640 1 -- 172.21.15.71:0/285496789 wait complete. 2024-03-25T19:47:48.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:48.367+0000 7fdeda94b640 1 librados: shutdown 2024-03-25T19:47:48.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:55.185 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:55.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.302+0000 7f7b5516c640 10 monclient: get_monmap_and_config 2024-03-25T19:47:55.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.302+0000 7f7b5516c640 10 monclient: build_initial_monmap 2024-03-25T19:47:55.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.302+0000 7f7b5516c640 10 monclient: monmap: 2024-03-25T19:47:55.302 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:55.302 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:55.303 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:47:55.303650+0000 2024-03-25T19:47:55.303 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:47:55.303650+0000 2024-03-25T19:47:55.303 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:55.303 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:55.303 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:55.303 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:47:55.303 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:47:55.303 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:55.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.303+0000 7f7b5516c640 1 Processor -- start 2024-03-25T19:47:55.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.303+0000 7f7b5516c640 1 -- start start 2024-03-25T19:47:55.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.303+0000 7f7b5516c640 10 monclient: init 2024-03-25T19:47:55.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 10 monclient(hunting): picked mon.noname-c con 0x7f7b500b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 10 monclient(hunting): picked mon.noname-a con 0x7f7b500b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 10 monclient(hunting): picked mon.noname-b con 0x7f7b5014be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b500af940 con 0x7f7b500b1730 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b500b0a70 con 0x7f7b500b4d60 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b500b0d50 con 0x7f7b5014be90 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:55.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.304+0000 7f7b5516c640 10 monclient(hunting): authenticate will time out at 5281.616480s 2024-03-25T19:47:55.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.313+0000 7f7b4ed9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7b500b4d60 0x7f7b5013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:33266/0 (socket says 172.21.15.71:33266) 2024-03-25T19:47:55.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.313+0000 7f7b4ed9d640 1 -- 172.21.15.71:0/896085615 learned_addr learned my addr 172.21.15.71:0/896085615 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:55.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.324+0000 7f7b4dd9b640 1 -- 172.21.15.71:0/896085615 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788924467 0 0) 0x7f7b500b0a70 con 0x7f7b500b4d60 2024-03-25T19:47:55.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.324+0000 7f7b4dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:47:55.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.324+0000 7f7b4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:55.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.324+0000 7f7b4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:55.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.324+0000 7f7b4dd9b640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:47:55.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.324+0000 7f7b4dd9b640 1 -- 172.21.15.71:0/896085615 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b38003520 con 0x7f7b500b4d60 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.332+0000 7f7b4dd9b640 1 -- 172.21.15.71:0/896085615 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 55251271 0 0) 0x7f7b38003520 con 0x7f7b500b4d60 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 1 -- 172.21.15.71:0/896085615 >> v1:172.21.15.71:6790/0 conn(0x7f7b5014be90 legacy=0x7f7b5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 1 -- 172.21.15.71:0/896085615 >> v1:172.21.15.71:6789/0 conn(0x7f7b500b1730 legacy=0x7f7b500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 1 monclient: found mon.noname-c 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 1 -- 172.21.15.71:0/896085615 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b500b1030 con 0x7f7b500b4d60 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 1 -- 172.21.15.71:0/896085615 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b40001690 con 0x7f7b500b4d60 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:55.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b4dd9b640 10 monclient: dump: 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b5516c640 5 monclient: authenticate success, global_id 4549 2024-03-25T19:47:55.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.333+0000 7f7b5516c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:55.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.338+0000 7f7b4dd9b640 1 -- 172.21.15.71:0/896085615 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b400017f0 con 0x7f7b500b4d60 2024-03-25T19:47:55.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.338+0000 7f7b4dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:55.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.338+0000 7f7b5516c640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:55.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.338+0000 7f7b5516c640 4 set_mon_vals no callback set 2024-03-25T19:47:55.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.339+0000 7f7b5516c640 10 monclient: shutdown 2024-03-25T19:47:55.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.339+0000 7f7b5516c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:55.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.339+0000 7f7b5516c640 1 -- 172.21.15.71:0/896085615 >> v1:172.21.15.193:6789/0 conn(0x7f7b500b4d60 legacy=0x7f7b5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:55.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.339+0000 7f7b5516c640 1 -- 172.21.15.71:0/896085615 shutdown_connections 2024-03-25T19:47:55.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.339+0000 7f7b5516c640 1 -- 172.21.15.71:0/896085615 >> 172.21.15.71:0/896085615 conn(0x7f7b500617a0 msgr2=0x7f7b50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:55.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.339+0000 7f7b5516c640 1 -- 172.21.15.71:0/896085615 shutdown_connections 2024-03-25T19:47:55.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.339+0000 7f7b5516c640 1 -- 172.21.15.71:0/896085615 wait complete. 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.340+0000 7f7b5516c640 10 monclient: build_initial_monmap 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.340+0000 7f7b5516c640 10 monclient: monmap: 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:47:55.340 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:55.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.341+0000 7f7b5516c640 1 librados: starting msgr at 2024-03-25T19:47:55.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.341+0000 7f7b5516c640 1 librados: starting objecter 2024-03-25T19:47:55.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.341+0000 7f7b5516c640 1 Processor -- start 2024-03-25T19:47:55.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.341+0000 7f7b5516c640 1 -- start start 2024-03-25T19:47:55.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.341+0000 7f7b5516c640 1 librados: setting wanted keys 2024-03-25T19:47:55.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.341+0000 7f7b5516c640 1 librados: calling monclient init 2024-03-25T19:47:55.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.341+0000 7f7b5516c640 10 monclient: init 2024-03-25T19:47:55.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 10 monclient(hunting): picked mon.noname-b con 0x7f7b500b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 10 monclient(hunting): picked mon.noname-c con 0x7f7b500b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5014be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b501d85b0 con 0x7f7b5014be90 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5013a8d0 con 0x7f7b500b1730 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b501d7980 con 0x7f7b500b4d60 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:55.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.342+0000 7f7b5516c640 10 monclient(hunting): authenticate will time out at 5281.654289s 2024-03-25T19:47:55.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.348+0000 7f7b4ed9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7b500b1730 0x7f7b501d7ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:33290/0 (socket says 172.21.15.71:33290) 2024-03-25T19:47:55.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.348+0000 7f7b4ed9d640 1 -- 172.21.15.71:0/1175043460 learned_addr learned my addr 172.21.15.71:0/1175043460 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:47:55.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.350+0000 7f7b377fe640 10 client.?.objecter ms_handle_connect 0x7f7b500b1730 2024-03-25T19:47:55.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.350+0000 7f7b377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:55.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.357+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327239729 0 0) 0x7f7b5013a8d0 con 0x7f7b500b1730 2024-03-25T19:47:55.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.357+0000 7f7b377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:55.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.357+0000 7f7b377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:55.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.357+0000 7f7b377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:55.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.357+0000 7f7b377fe640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:47:55.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.357+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b3c004eb0 con 0x7f7b500b1730 2024-03-25T19:47:55.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.363+0000 7f7b377fe640 10 client.?.objecter ms_handle_connect 0x7f7b5014be90 2024-03-25T19:47:55.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.363+0000 7f7b377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:55.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.366+0000 7f7b377fe640 10 client.?.objecter ms_handle_connect 0x7f7b500b4d60 2024-03-25T19:47:55.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.366+0000 7f7b377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.369+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821324511 0 0) 0x7f7b3c004eb0 con 0x7f7b500b1730 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.369+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b5013a8d0 con 0x7f7b500b1730 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.369+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b40001d60 con 0x7f7b500b1730 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.369+0000 7f7b377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.369+0000 7f7b377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.369+0000 7f7b377fe640 10 monclient(hunting): dump: 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:55.369 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:55.370 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:55.370 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:55.370 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:55.370 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:55.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.369+0000 7f7b377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4089853765 0 0) 0x7f7b5013a8d0 con 0x7f7b500b1730 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 >> v1:172.21.15.71:6790/0 conn(0x7f7b500b4d60 legacy=0x7f7b5013b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 >> v1:172.21.15.71:6789/0 conn(0x7f7b5014be90 legacy=0x7f7b501e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 1 monclient: found mon.b 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50139a30 con 0x7f7b500b1730 2024-03-25T19:47:55.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 5 monclient: authenticate success, global_id 4552 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 10 monclient: _renew_subs 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5013bf90 con 0x7f7b500b1730 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 10 monclient: _renew_subs 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b501391e0 con 0x7f7b500b1730 2024-03-25T19:47:55.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.379+0000 7f7b5516c640 1 librados: init done 2024-03-25T19:47:55.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.380+0000 7f7b5516c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:55.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.380+0000 7f7b5516c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:55.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.380+0000 7f7b5516c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:55.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.380+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b501d77f0 con 0x7f7b500b1730 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.662+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b40004a90 con 0x7f7b500b1730 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.662+0000 7f7b377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.662+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7b40004ea0 con 0x7f7b500b1730 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.662+0000 7f7b377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.662+0000 7f7b377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.662+0000 7f7b377fe640 10 monclient: dump: 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:47:55.662 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:55.663 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:47:55.663 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:55.663 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:47:55.663 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:47:55.663 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:47:55.663 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:47:55.663 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:47:55.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.662+0000 7f7b4d59a640 4 set_mon_vals no callback set 2024-03-25T19:47:55.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.663+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f7b40005080 con 0x7f7b500b1730 2024-03-25T19:47:55.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.664+0000 7f7b4e59c640 10 monclient: get_auth_request con 0x7f7b3c06dc30 auth_method 0 2024-03-25T19:47:55.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.664+0000 7f7b377fe640 10 client.4552.objecter ms_handle_connect 0x7f7b3c06dc30 2024-03-25T19:47:55.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.670+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f7b40054a70 con 0x7f7b500b1730 2024-03-25T19:47:55.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.671+0000 7f7b377fe640 10 client.4552.objecter ms_dispatch 0x7f7b50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:55.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.671+0000 7f7b377fe640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:55.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.671+0000 7f7b377fe640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:55.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.671+0000 7f7b377fe640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T19:47:55.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.671+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b40050580 con 0x7f7b500b1730 2024-03-25T19:47:55.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.671+0000 7f7b377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:55.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.671+0000 7f7b377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:55.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.926+0000 7f7b5516c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:55.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.926+0000 7f7b5516c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:55.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.926+0000 7f7b5516c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:47:55.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:55.926+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b50139bc0 con 0x7f7b500b1730 2024-03-25T19:47:56.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.064+0000 7f7b377fe640 1 -- 172.21.15.71:0/1175043460 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f7b400524b0 con 0x7f7b500b1730 2024-03-25T19:47:56.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.064+0000 7f7b377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:56.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.064+0000 7f7b377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:56.064 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:47:56.064 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 10 librados: watch_flush enter 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 10 librados: watch_flush exit 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 10 client.4552.objecter successfully canceled tick 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 >> v1:172.21.15.71:6800/3911735624 conn(0x7f7b3c06dc30 legacy=0x7f7b3c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 10 monclient: shutdown 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 >> v1:172.21.15.193:6789/0 conn(0x7f7b500b1730 legacy=0x7f7b501d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 shutdown_connections 2024-03-25T19:47:56.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 >> 172.21.15.71:0/1175043460 conn(0x7f7b500617a0 msgr2=0x7f7b500b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:56.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 shutdown_connections 2024-03-25T19:47:56.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 1 -- 172.21.15.71:0/1175043460 wait complete. 2024-03-25T19:47:56.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:47:56.066+0000 7f7b5516c640 1 librados: shutdown 2024-03-25T19:47:56.079 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:02.609 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:45.365 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.623 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:45.380 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.661 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:07:45.417 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.675 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:45.432 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.679 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:45.398 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.681 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:45.437 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.687 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:45.406 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.709 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:07:45.466 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.722 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:45.441 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.886 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:02.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.986+0000 7f1e2a24a640 10 monclient: get_monmap_and_config 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.986+0000 7f1e2a24a640 10 monclient: build_initial_monmap 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.986+0000 7f1e2a24a640 10 monclient: monmap: 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:48:02.987007+0000 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:48:02.987007+0000 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:02.986 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:48:02.987 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:48:02.987 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.987+0000 7f1e2a24a640 1 Processor -- start 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.987+0000 7f1e2a24a640 1 -- start start 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.987+0000 7f1e2a24a640 10 monclient: init 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 10 monclient(hunting): picked mon.noname-a con 0x7f1e240a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 10 monclient(hunting): picked mon.noname-b con 0x7f1e2413f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 10 monclient(hunting): picked mon.noname-c con 0x7f1e241430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:02.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e24142520 con 0x7f1e240a98f0 2024-03-25T19:48:02.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e240b28a0 con 0x7f1e241430c0 2024-03-25T19:48:02.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e240b2a30 con 0x7f1e2413f710 2024-03-25T19:48:02.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:02.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.988+0000 7f1e2a24a640 10 monclient(hunting): authenticate will time out at 5289.300368s 2024-03-25T19:48:02.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.994+0000 7f1e23fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f1e241430c0 0x7f1e2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:51104/0 (socket says 172.21.15.71:51104) 2024-03-25T19:48:02.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.994+0000 7f1e23fff640 1 -- 172.21.15.71:0/1127080727 learned_addr learned my addr 172.21.15.71:0/1127080727 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:02.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.998+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681764983 0 0) 0x7f1e240b28a0 con 0x7f1e241430c0 2024-03-25T19:48:02.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.998+0000 7f1e227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:02.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.998+0000 7f1e227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:02.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.998+0000 7f1e227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:02.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.998+0000 7f1e227fc640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:48:02.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:02.998+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e0c003520 con 0x7f1e241430c0 2024-03-25T19:48:03.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854038293 0 0) 0x7f1e0c003520 con 0x7f1e241430c0 2024-03-25T19:48:03.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 >> v1:172.21.15.71:6790/0 conn(0x7f1e2413f710 legacy=0x7f1e241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:48:03.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 >> v1:172.21.15.71:6789/0 conn(0x7f1e240a98f0 legacy=0x7f1e240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:48:03.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:03.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 1 monclient: found mon.noname-c 2024-03-25T19:48:03.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:03.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:48:03.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e240b44c0 con 0x7f1e241430c0 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1e1c001a10 con 0x7f1e241430c0 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e2a24a640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e2a24a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e227fc640 10 monclient: dump: 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:03.009 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:03.010 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:03.010 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:03.010 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:03.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.008+0000 7f1e2a24a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.021+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e1c001be0 con 0x7f1e241430c0 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.021+0000 7f1e227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.021+0000 7f1e227fc640 1 -- 172.21.15.71:0/1127080727 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1e1c002030 con 0x7f1e241430c0 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.021+0000 7f1e2a24a640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.021+0000 7f1e2a24a640 4 set_mon_vals no callback set 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.021+0000 7f1e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.021+0000 7f1e227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.021+0000 7f1e227fc640 10 monclient: dump: 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:03.021 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:03.022 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:03.022 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:03.022 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:03.022 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:03.022 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:03.022 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:03.022 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:03.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.022+0000 7f1e2a24a640 10 monclient: shutdown 2024-03-25T19:48:03.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.022+0000 7f1e2a24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:03.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.022+0000 7f1e2a24a640 1 -- 172.21.15.71:0/1127080727 >> v1:172.21.15.193:6789/0 conn(0x7f1e241430c0 legacy=0x7f1e2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.022+0000 7f1e2a24a640 1 -- 172.21.15.71:0/1127080727 shutdown_connections 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.022+0000 7f1e2a24a640 1 -- 172.21.15.71:0/1127080727 >> 172.21.15.71:0/1127080727 conn(0x7f1e240b7ea0 msgr2=0x7f1e240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.022+0000 7f1e2a24a640 1 -- 172.21.15.71:0/1127080727 shutdown_connections 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.022+0000 7f1e2a24a640 1 -- 172.21.15.71:0/1127080727 wait complete. 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.023+0000 7f1e2a24a640 10 monclient: build_initial_monmap 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.023+0000 7f1e2a24a640 10 monclient: monmap: 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:03.024 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.023+0000 7f1e2a24a640 1 librados: starting msgr at 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.023+0000 7f1e2a24a640 1 librados: starting objecter 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.023+0000 7f1e2a24a640 1 Processor -- start 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 1 -- start start 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 1 librados: setting wanted keys 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 1 librados: calling monclient init 2024-03-25T19:48:03.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 10 monclient: init 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 10 monclient(hunting): picked mon.noname-b con 0x7f1e240a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 10 monclient(hunting): picked mon.noname-a con 0x7f1e2413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 10 monclient(hunting): picked mon.noname-c con 0x7f1e241430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e241384b0 con 0x7f1e2413f710 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e241394e0 con 0x7f1e240a98f0 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e24139750 con 0x7f1e241430c0 2024-03-25T19:48:03.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:03.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.024+0000 7f1e2a24a640 10 monclient(hunting): authenticate will time out at 5289.336565s 2024-03-25T19:48:03.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.030+0000 7f1e23fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f1e241430c0 0x7f1e241e1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:40814/0 (socket says 172.21.15.71:40814) 2024-03-25T19:48:03.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.030+0000 7f1e23fff640 1 -- 172.21.15.71:0/3431316644 learned_addr learned my addr 172.21.15.71:0/3431316644 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:03.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.031+0000 7f1e0bfff640 10 client.?.objecter ms_handle_connect 0x7f1e241430c0 2024-03-25T19:48:03.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.031+0000 7f1e0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:03.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.043+0000 7f1e0bfff640 10 client.?.objecter ms_handle_connect 0x7f1e2413f710 2024-03-25T19:48:03.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.043+0000 7f1e0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:03.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.047+0000 7f1e0bfff640 10 client.?.objecter ms_handle_connect 0x7f1e240a98f0 2024-03-25T19:48:03.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.047+0000 7f1e0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:03.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.056+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022100383 0 0) 0x7f1e24139750 con 0x7f1e241430c0 2024-03-25T19:48:03.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.057+0000 7f1e0bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:03.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.057+0000 7f1e0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:03.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.057+0000 7f1e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:03.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.057+0000 7f1e0bfff640 10 monclient(hunting): my global_id is 14558 2024-03-25T19:48:03.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.057+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e10004eb0 con 0x7f1e241430c0 2024-03-25T19:48:03.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.063+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420777916 0 0) 0x7f1e241384b0 con 0x7f1e2413f710 2024-03-25T19:48:03.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.063+0000 7f1e0bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:03.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.063+0000 7f1e0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:03.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.063+0000 7f1e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:03.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.063+0000 7f1e0bfff640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:48:03.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.063+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e24139750 con 0x7f1e2413f710 2024-03-25T19:48:03.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.067+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4249681717 0 0) 0x7f1e10004eb0 con 0x7f1e241430c0 2024-03-25T19:48:03.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.067+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e241384b0 con 0x7f1e241430c0 2024-03-25T19:48:03.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.067+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1e1c001f30 con 0x7f1e241430c0 2024-03-25T19:48:03.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.067+0000 7f1e0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:03.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.067+0000 7f1e0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:03.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.067+0000 7f1e0bfff640 10 monclient(hunting): dump: 2024-03-25T19:48:03.067 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:03.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.067+0000 7f1e0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.077+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2161295410 0 0) 0x7f1e241384b0 con 0x7f1e241430c0 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 >> v1:172.21.15.193:6789/0 conn(0x7f1e240a98f0 legacy=0x7f1e24137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 >> v1:172.21.15.71:6789/0 conn(0x7f1e2413f710 legacy=0x7f1e24134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 1 monclient: found mon.c 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e24138940 con 0x7f1e241430c0 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 10 monclient: _finish_auth 0 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:03.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 5 monclient: authenticate success, global_id 14558 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 10 monclient: _renew_subs 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e24137e70 con 0x7f1e241430c0 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 10 client.14558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 10 monclient: _renew_subs 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e241356d0 con 0x7f1e241430c0 2024-03-25T19:48:03.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.078+0000 7f1e2a24a640 1 librados: init done 2024-03-25T19:48:03.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.079+0000 7f1e2a24a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:03.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.079+0000 7f1e2a24a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:03.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.079+0000 7f1e2a24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:03.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.079+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e240627b0 con 0x7f1e241430c0 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.386+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e1c004e10 con 0x7f1e241430c0 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.386+0000 7f1e0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.386+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1e1c0015f0 con 0x7f1e241430c0 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.386+0000 7f1e0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.386+0000 7f1e0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.386+0000 7f1e0bfff640 10 monclient: dump: 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:03.386 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:03.387 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:03.387 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:03.387 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:03.387 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:03.387 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:03.387 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:03.387 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:03.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.386+0000 7f1e21ffb640 4 set_mon_vals no callback set 2024-03-25T19:48:03.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.387+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f1e1c01afd0 con 0x7f1e241430c0 2024-03-25T19:48:03.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.387+0000 7f1e237fe640 10 monclient: get_auth_request con 0x7f1e1006dd10 auth_method 0 2024-03-25T19:48:03.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.388+0000 7f1e0bfff640 10 client.14558.objecter ms_handle_connect 0x7f1e1006dd10 2024-03-25T19:48:03.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.393+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f1e1c054bc0 con 0x7f1e241430c0 2024-03-25T19:48:03.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.393+0000 7f1e0bfff640 10 client.14558.objecter ms_dispatch 0x7f1e24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:03.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.393+0000 7f1e0bfff640 3 client.14558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:03.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.393+0000 7f1e0bfff640 3 client.14558.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:03.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.393+0000 7f1e0bfff640 20 client.14558.objecter dump_active .. 0 homeless 2024-03-25T19:48:03.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.393+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e1c050500 con 0x7f1e241430c0 2024-03-25T19:48:03.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.393+0000 7f1e0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:03.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.393+0000 7f1e0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:03.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.642+0000 7f1e2a24a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:03.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.642+0000 7f1e2a24a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:03.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.642+0000 7f1e2a24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:03.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.642+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e24138660 con 0x7f1e241430c0 2024-03-25T19:48:03.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.669+0000 7f1e0bfff640 1 -- 172.21.15.71:0/3431316644 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f1e24138660 con 0x7f1e241430c0 2024-03-25T19:48:03.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.669+0000 7f1e0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:03.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.669+0000 7f1e0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:03.669 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:48:03.669 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:03.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 10 librados: watch_flush enter 2024-03-25T19:48:03.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 10 librados: watch_flush exit 2024-03-25T19:48:03.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 20 client.14558.objecter shutdown clearing up homeless session... 2024-03-25T19:48:03.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 10 client.14558.objecter successfully canceled tick 2024-03-25T19:48:03.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 >> v1:172.21.15.71:6800/3911735624 conn(0x7f1e1006dd10 legacy=0x7f1e100701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:03.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 10 monclient: shutdown 2024-03-25T19:48:03.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:03.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 >> v1:172.21.15.71:6790/0 conn(0x7f1e241430c0 legacy=0x7f1e241e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:03.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 shutdown_connections 2024-03-25T19:48:03.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 >> 172.21.15.71:0/3431316644 conn(0x7f1e240b7ea0 msgr2=0x7f1e24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:03.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 shutdown_connections 2024-03-25T19:48:03.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 1 -- 172.21.15.71:0/3431316644 wait complete. 2024-03-25T19:48:03.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:03.671+0000 7f1e2a24a640 1 librados: shutdown 2024-03-25T19:48:03.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:10.490 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.604+0000 7f348956d640 10 monclient: get_monmap_and_config 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.604+0000 7f348956d640 10 monclient: build_initial_monmap 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.604+0000 7f348956d640 10 monclient: monmap: 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:48:10.605226+0000 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:48:10.605226+0000 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:10.604 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:48:10.605 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:48:10.605 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:10.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.604+0000 7f348956d640 1 Processor -- start 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.604+0000 7f348956d640 1 -- start start 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.604+0000 7f348956d640 10 monclient: init 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 10 monclient(hunting): picked mon.noname-b con 0x7f348406c380 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 10 monclient(hunting): picked mon.noname-a con 0x7f348406d670 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 10 monclient(hunting): picked mon.noname-c con 0x7f34840af8f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348406c9a0 con 0x7f348406d670 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840ada90 con 0x7f34840af8f0 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840b0d10 con 0x7f348406c380 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:10.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.605+0000 7f348956d640 10 monclient(hunting): authenticate will time out at 5296.917430s 2024-03-25T19:48:10.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.615+0000 7f34837fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f34840af8f0 0x7f34840aedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:33142/0 (socket says 172.21.15.71:33142) 2024-03-25T19:48:10.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.615+0000 7f34837fe640 1 -- 172.21.15.71:0/197934188 learned_addr learned my addr 172.21.15.71:0/197934188 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:10.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.622+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802055784 0 0) 0x7f34840ada90 con 0x7f34840af8f0 2024-03-25T19:48:10.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.622+0000 7f3481ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:10.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.622+0000 7f3481ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:10.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.622+0000 7f3481ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:10.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.622+0000 7f3481ffb640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:48:10.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.622+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346c003520 con 0x7f34840af8f0 2024-03-25T19:48:10.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.629+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991679364 0 0) 0x7f34840b0d10 con 0x7f348406c380 2024-03-25T19:48:10.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.629+0000 7f3481ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:10.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.629+0000 7f3481ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:10.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.629+0000 7f3481ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:10.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.629+0000 7f3481ffb640 10 monclient(hunting): my global_id is 14561 2024-03-25T19:48:10.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.629+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34840ada90 con 0x7f348406c380 2024-03-25T19:48:10.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.630+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193843604 0 0) 0x7f348406c9a0 con 0x7f348406d670 2024-03-25T19:48:10.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.630+0000 7f3481ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:10.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.630+0000 7f3481ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:10.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.630+0000 7f3481ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:10.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.630+0000 7f3481ffb640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:48:10.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.630+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34840b0d10 con 0x7f348406d670 2024-03-25T19:48:10.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 742916311 0 0) 0x7f346c003520 con 0x7f34840af8f0 2024-03-25T19:48:10.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 >> v1:172.21.15.71:6790/0 conn(0x7f348406c380 legacy=0x7f34840b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 >> v1:172.21.15.71:6789/0 conn(0x7f348406d670 legacy=0x7f34840ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 1 monclient: found mon.noname-c 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34840add00 con 0x7f34840af8f0 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f347c001a10 con 0x7f34840af8f0 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:10.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f3481ffb640 10 monclient: dump: 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f348956d640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:48:10.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.644+0000 7f348956d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.659+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f347c001b70 con 0x7f34840af8f0 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.659+0000 7f3481ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.659+0000 7f3481ffb640 1 -- 172.21.15.71:0/197934188 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f347c001f00 con 0x7f34840af8f0 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.659+0000 7f3481ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.659+0000 7f3481ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.659+0000 7f3481ffb640 10 monclient: dump: 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:10.659 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:10.660 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:10.660 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:10.660 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:10.660 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:10.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.659+0000 7f348956d640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:10.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.659+0000 7f348956d640 4 set_mon_vals no callback set 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.660+0000 7f348956d640 10 monclient: shutdown 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.660+0000 7f348956d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.660+0000 7f348956d640 1 -- 172.21.15.71:0/197934188 >> v1:172.21.15.193:6789/0 conn(0x7f34840af8f0 legacy=0x7f34840aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.660+0000 7f348956d640 1 -- 172.21.15.71:0/197934188 shutdown_connections 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.660+0000 7f348956d640 1 -- 172.21.15.71:0/197934188 >> 172.21.15.71:0/197934188 conn(0x7f34840b5c00 msgr2=0x7f34840b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.660+0000 7f348956d640 1 -- 172.21.15.71:0/197934188 shutdown_connections 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.660+0000 7f348956d640 1 -- 172.21.15.71:0/197934188 wait complete. 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.661+0000 7f348956d640 10 monclient: build_initial_monmap 2024-03-25T19:48:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.661+0000 7f348956d640 10 monclient: monmap: 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.661+0000 7f348956d640 1 librados: starting msgr at 2024-03-25T19:48:10.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.661+0000 7f348956d640 1 librados: starting objecter 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.662+0000 7f348956d640 1 Processor -- start 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.662+0000 7f348956d640 1 -- start start 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.662+0000 7f348956d640 1 librados: setting wanted keys 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.662+0000 7f348956d640 1 librados: calling monclient init 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.662+0000 7f348956d640 10 monclient: init 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 10 monclient(hunting): picked mon.noname-c con 0x7f348406c380 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 10 monclient(hunting): picked mon.noname-a con 0x7f348406d670 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 10 monclient(hunting): picked mon.noname-b con 0x7f34840af8f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34841451e0 con 0x7f348406d670 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3484143b80 con 0x7f34840af8f0 2024-03-25T19:48:10.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34841db5d0 con 0x7f348406c380 2024-03-25T19:48:10.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:10.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.663+0000 7f348956d640 10 monclient(hunting): authenticate will time out at 5296.975162s 2024-03-25T19:48:10.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.666+0000 7f34827fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f348406d670 0x7f348413aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:45694/0 (socket says 172.21.15.71:45694) 2024-03-25T19:48:10.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.666+0000 7f34827fc640 1 -- 172.21.15.71:0/2976287004 learned_addr learned my addr 172.21.15.71:0/2976287004 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:10.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.667+0000 7f346b7fe640 10 client.?.objecter ms_handle_connect 0x7f348406d670 2024-03-25T19:48:10.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.667+0000 7f346b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:10.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.681+0000 7f346b7fe640 10 client.?.objecter ms_handle_connect 0x7f348406c380 2024-03-25T19:48:10.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.681+0000 7f346b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:10.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.681+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560121062 0 0) 0x7f34841451e0 con 0x7f348406d670 2024-03-25T19:48:10.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.681+0000 7f346b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:10.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.681+0000 7f346b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:10.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.681+0000 7f346b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:10.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.681+0000 7f346b7fe640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:48:10.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.681+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3470004eb0 con 0x7f348406d670 2024-03-25T19:48:10.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.690+0000 7f346b7fe640 10 client.?.objecter ms_handle_connect 0x7f34840af8f0 2024-03-25T19:48:10.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.690+0000 7f346b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.691+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250410102 0 0) 0x7f3470004eb0 con 0x7f348406d670 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.691+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34841451e0 con 0x7f348406d670 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.691+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f34780019d0 con 0x7f348406d670 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.691+0000 7f346b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.691+0000 7f346b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.691+0000 7f346b7fe640 10 monclient(hunting): dump: 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:10.691 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:10.692 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:10.692 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:10.692 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:10.692 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:10.692 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:10.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.691+0000 7f346b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:10.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.696+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864045713 0 0) 0x7f34841db5d0 con 0x7f348406c380 2024-03-25T19:48:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.696+0000 7f346b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.696+0000 7f346b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.696+0000 7f346b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.696+0000 7f346b7fe640 10 monclient(hunting): my global_id is 14564 2024-03-25T19:48:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.696+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3470004eb0 con 0x7f348406c380 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3289594373 0 0) 0x7f34841451e0 con 0x7f348406d670 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 >> v1:172.21.15.71:6790/0 conn(0x7f348406c380 legacy=0x7f34841daec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 >> v1:172.21.15.193:6789/0 conn(0x7f34840af8f0 legacy=0x7f34841e3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 1 monclient: found mon.a 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3484147570 con 0x7f348406d670 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:10.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f346b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 5 monclient: authenticate success, global_id 4566 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 10 monclient: _renew_subs 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34841477c0 con 0x7f348406d670 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 10 monclient: _renew_subs 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3484143840 con 0x7f348406d670 2024-03-25T19:48:10.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.701+0000 7f348956d640 1 librados: init done 2024-03-25T19:48:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.702+0000 7f348956d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.702+0000 7f348956d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.702+0000 7f348956d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:48:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.702+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3484143d60 con 0x7f348406d670 2024-03-25T19:48:10.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.915+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3478001c50 con 0x7f348406d670 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.915+0000 7f346b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.915+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3478003a60 con 0x7f348406d670 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.915+0000 7f346b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.915+0000 7f346b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.915+0000 7f346b7fe640 10 monclient: dump: 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:10.915 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:10.916 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:10.916 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:10.916 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:10.915+0000 7f34817fa640 4 set_mon_vals no callback set 2024-03-25T19:48:11.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.004+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f3478003bc0 con 0x7f348406d670 2024-03-25T19:48:11.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.004+0000 7f3482ffd640 10 monclient: get_auth_request con 0x7f347006dd90 auth_method 0 2024-03-25T19:48:11.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.005+0000 7f346b7fe640 10 client.4566.objecter ms_handle_connect 0x7f347006dd90 2024-03-25T19:48:11.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.015+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f3478054820 con 0x7f348406d670 2024-03-25T19:48:11.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.015+0000 7f346b7fe640 10 client.4566.objecter ms_dispatch 0x7f3484000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:11.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.015+0000 7f346b7fe640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:11.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.015+0000 7f346b7fe640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:11.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.015+0000 7f346b7fe640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-25T19:48:11.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.015+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3478050160 con 0x7f348406d670 2024-03-25T19:48:11.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.015+0000 7f346b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:11.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.015+0000 7f346b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:11.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.236+0000 7f348956d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:11.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.236+0000 7f348956d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:11.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.236+0000 7f348956d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:48:11.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.236+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34840627b0 con 0x7f348406d670 2024-03-25T19:48:11.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.264+0000 7f346b7fe640 1 -- 172.21.15.71:0/2976287004 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f3478052260 con 0x7f348406d670 2024-03-25T19:48:11.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.264+0000 7f346b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:11.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.264+0000 7f346b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:11.264 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:48:11.264 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 10 librados: watch_flush enter 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 10 librados: watch_flush exit 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 10 client.4566.objecter successfully canceled tick 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 >> v1:172.21.15.71:6800/3911735624 conn(0x7f347006dd90 legacy=0x7f3470070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 10 monclient: shutdown 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 >> v1:172.21.15.71:6789/0 conn(0x7f348406d670 legacy=0x7f348413aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 shutdown_connections 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 >> 172.21.15.71:0/2976287004 conn(0x7f34840b5c00 msgr2=0x7f3484148510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.266+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 shutdown_connections 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.267+0000 7f348956d640 1 -- 172.21.15.71:0/2976287004 wait complete. 2024-03-25T19:48:11.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:11.267+0000 7f348956d640 1 librados: shutdown 2024-03-25T19:48:11.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:18.084 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:18.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.202+0000 7f3a077c6640 10 monclient: get_monmap_and_config 2024-03-25T19:48:18.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.202+0000 7f3a077c6640 10 monclient: build_initial_monmap 2024-03-25T19:48:18.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.202+0000 7f3a077c6640 10 monclient: monmap: 2024-03-25T19:48:18.202 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:18.202 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:18.202 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:48:18.203474+0000 2024-03-25T19:48:18.202 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:48:18.203474+0000 2024-03-25T19:48:18.203 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:18.203 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:18.203 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:18.203 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:48:18.203 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:48:18.203 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.203+0000 7f3a077c6640 1 Processor -- start 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.203+0000 7f3a077c6640 1 -- start start 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.203+0000 7f3a077c6640 10 monclient: init 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3a000a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3a001430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:18.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a00142520 con 0x7f3a0013f710 2024-03-25T19:48:18.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a000b28a0 con 0x7f3a001430c0 2024-03-25T19:48:18.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a000b2a30 con 0x7f3a000a98f0 2024-03-25T19:48:18.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:18.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.204+0000 7f3a077c6640 10 monclient(hunting): authenticate will time out at 5304.516425s 2024-03-25T19:48:18.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.213+0000 7f3a05562640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3a000a98f0 0x7f3a000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:40996/0 (socket says 172.21.15.71:40996) 2024-03-25T19:48:18.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.213+0000 7f3a05562640 1 -- 172.21.15.71:0/969527793 learned_addr learned my addr 172.21.15.71:0/969527793 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:18.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.228+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918536967 0 0) 0x7f3a000b28a0 con 0x7f3a001430c0 2024-03-25T19:48:18.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.228+0000 7f39f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:18.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.228+0000 7f39f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:18.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.228+0000 7f39f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:18.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.228+0000 7f39f7fff640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:48:18.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.228+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ec003520 con 0x7f3a001430c0 2024-03-25T19:48:18.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.230+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560254756 0 0) 0x7f3a000b2a30 con 0x7f3a000a98f0 2024-03-25T19:48:18.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.230+0000 7f39f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:18.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.230+0000 7f39f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:18.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.230+0000 7f39f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:18.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.230+0000 7f39f7fff640 10 monclient(hunting): my global_id is 14567 2024-03-25T19:48:18.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.230+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a000b28a0 con 0x7f3a000a98f0 2024-03-25T19:48:18.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.234+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413478680 0 0) 0x7f3a00142520 con 0x7f3a0013f710 2024-03-25T19:48:18.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.234+0000 7f39f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:18.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.234+0000 7f39f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:18.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.234+0000 7f39f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:18.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.234+0000 7f39f7fff640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:48:18.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.234+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a000b2a30 con 0x7f3a0013f710 2024-03-25T19:48:18.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2442319383 0 0) 0x7f39ec003520 con 0x7f3a001430c0 2024-03-25T19:48:18.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 >> v1:172.21.15.71:6790/0 conn(0x7f3a000a98f0 legacy=0x7f3a000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:18.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 >> v1:172.21.15.71:6789/0 conn(0x7f3a0013f710 legacy=0x7f3a001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:18.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:18.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 1 monclient: found mon.noname-c 2024-03-25T19:48:18.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:18.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a000b44c0 con 0x7f3a001430c0 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 10 monclient: _finish_auth 0 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f39f8001a10 con 0x7f3a001430c0 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f39f7fff640 10 monclient: dump: 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:18.246 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f3a077c6640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:48:18.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.245+0000 7f3a077c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.250+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39f8001b70 con 0x7f3a001430c0 2024-03-25T19:48:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.250+0000 7f39f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.250+0000 7f39f7fff640 1 -- 172.21.15.71:0/969527793 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f39f8001f80 con 0x7f3a001430c0 2024-03-25T19:48:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.250+0000 7f39f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.250+0000 7f39f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.250+0000 7f39f7fff640 10 monclient: dump: 2024-03-25T19:48:18.250 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.250+0000 7f3a077c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:18.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.250+0000 7f3a077c6640 4 set_mon_vals no callback set 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.251+0000 7f3a077c6640 10 monclient: shutdown 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.251+0000 7f3a077c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.251+0000 7f3a077c6640 1 -- 172.21.15.71:0/969527793 >> v1:172.21.15.193:6789/0 conn(0x7f3a001430c0 legacy=0x7f3a0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.251+0000 7f3a077c6640 1 -- 172.21.15.71:0/969527793 shutdown_connections 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.251+0000 7f3a077c6640 1 -- 172.21.15.71:0/969527793 >> 172.21.15.71:0/969527793 conn(0x7f3a000b7ea0 msgr2=0x7f3a000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.251+0000 7f3a077c6640 1 -- 172.21.15.71:0/969527793 shutdown_connections 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.251+0000 7f3a077c6640 1 -- 172.21.15.71:0/969527793 wait complete. 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.252+0000 7f3a077c6640 10 monclient: build_initial_monmap 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.252+0000 7f3a077c6640 10 monclient: monmap: 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:48:18.254 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.252+0000 7f3a077c6640 1 librados: starting msgr at 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.252+0000 7f3a077c6640 1 librados: starting objecter 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 1 Processor -- start 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 1 -- start start 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 1 librados: setting wanted keys 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 1 librados: calling monclient init 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 10 monclient: init 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3a000a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3a001430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a00135230 con 0x7f3a0013f710 2024-03-25T19:48:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a00135da0 con 0x7f3a001430c0 2024-03-25T19:48:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a00135f30 con 0x7f3a000a98f0 2024-03-25T19:48:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.253+0000 7f3a077c6640 10 monclient(hunting): authenticate will time out at 5304.565830s 2024-03-25T19:48:18.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.262+0000 7f3a05562640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3a000a98f0 0x7f3a001d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:41028/0 (socket says 172.21.15.71:41028) 2024-03-25T19:48:18.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.262+0000 7f3a05562640 1 -- 172.21.15.71:0/834357704 learned_addr learned my addr 172.21.15.71:0/834357704 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:18.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.265+0000 7f39f5ffb640 10 client.?.objecter ms_handle_connect 0x7f3a000a98f0 2024-03-25T19:48:18.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.265+0000 7f39f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.271+0000 7f39f5ffb640 10 client.?.objecter ms_handle_connect 0x7f3a0013f710 2024-03-25T19:48:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.271+0000 7f39f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.271+0000 7f39f5ffb640 10 client.?.objecter ms_handle_connect 0x7f3a001430c0 2024-03-25T19:48:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.271+0000 7f39f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.276+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705252884 0 0) 0x7f3a00135f30 con 0x7f3a000a98f0 2024-03-25T19:48:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.276+0000 7f39f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.276+0000 7f39f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.277+0000 7f39f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:18.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.277+0000 7f39f5ffb640 10 monclient(hunting): my global_id is 14570 2024-03-25T19:48:18.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.277+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39e4004eb0 con 0x7f3a000a98f0 2024-03-25T19:48:18.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.280+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835367387 0 0) 0x7f3a00135230 con 0x7f3a0013f710 2024-03-25T19:48:18.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.280+0000 7f39f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:18.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.280+0000 7f39f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:18.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.280+0000 7f39f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:18.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.280+0000 7f39f5ffb640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:48:18.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.280+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a00135f30 con 0x7f3a0013f710 2024-03-25T19:48:18.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.284+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498154662 0 0) 0x7f3a00135da0 con 0x7f3a001430c0 2024-03-25T19:48:18.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.285+0000 7f39f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:18.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.285+0000 7f39f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:18.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.285+0000 7f39f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:18.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.285+0000 7f39f5ffb640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:48:18.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.285+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a00135230 con 0x7f3a001430c0 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.290+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2327389043 0 0) 0x7f3a00135f30 con 0x7f3a0013f710 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.290+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a00135da0 con 0x7f3a0013f710 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.290+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f39f0001870 con 0x7f3a0013f710 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.290+0000 7f39f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.290+0000 7f39f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.290+0000 7f39f5ffb640 10 monclient(hunting): dump: 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:18.290 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:18.291 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:18.291 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:18.291 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:18.291 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:18.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.290+0000 7f39f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.295+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3183959163 0 0) 0x7f39e4004eb0 con 0x7f3a000a98f0 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.295+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a00135f30 con 0x7f3a000a98f0 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.295+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f39fc0019f0 con 0x7f3a000a98f0 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.295+0000 7f39f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.295+0000 7f39f5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.295+0000 7f39f5ffb640 10 monclient(hunting): dump: 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:18.295 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:18.296 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:18.296 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:18.296 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:18.296 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:18.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 415470804 0 0) 0x7f3a00135f30 con 0x7f3a000a98f0 2024-03-25T19:48:18.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 >> v1:172.21.15.193:6789/0 conn(0x7f3a001430c0 legacy=0x7f3a001db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:18.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 >> v1:172.21.15.71:6789/0 conn(0x7f3a0013f710 legacy=0x7f3a001dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:18.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:18.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 1 monclient: found mon.c 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a001360c0 con 0x7f3a000a98f0 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f39f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 5 monclient: authenticate success, global_id 14570 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 10 monclient: _renew_subs 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a001d5000 con 0x7f3a000a98f0 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 10 client.14570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:18.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 10 monclient: _renew_subs 2024-03-25T19:48:18.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:18.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a001d41e0 con 0x7f3a000a98f0 2024-03-25T19:48:18.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.303+0000 7f3a077c6640 1 librados: init done 2024-03-25T19:48:18.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.304+0000 7f3a077c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:18.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.304+0000 7f3a077c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:18.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.304+0000 7f3a077c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:18.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.304+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a00135090 con 0x7f3a000a98f0 2024-03-25T19:48:18.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.604+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39fc0047e0 con 0x7f3a000a98f0 2024-03-25T19:48:18.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.604+0000 7f39f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:18.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.604+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f39fc004950 con 0x7f3a000a98f0 2024-03-25T19:48:18.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.604+0000 7f39f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:18.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.604+0000 7f39f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:18.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.604+0000 7f39f5ffb640 10 monclient: dump: 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:18.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.604+0000 7f39f77fe640 4 set_mon_vals no callback set 2024-03-25T19:48:18.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.605+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f39fc01a9a0 con 0x7f3a000a98f0 2024-03-25T19:48:18.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.605+0000 7f3a04d61640 10 monclient: get_auth_request con 0x7f39e406dfb0 auth_method 0 2024-03-25T19:48:18.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.606+0000 7f39f5ffb640 10 client.14570.objecter ms_handle_connect 0x7f39e406dfb0 2024-03-25T19:48:18.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.611+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f39fc0546c0 con 0x7f3a000a98f0 2024-03-25T19:48:18.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.611+0000 7f39f5ffb640 10 client.14570.objecter ms_dispatch 0x7f3a00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.611+0000 7f39f5ffb640 3 client.14570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.611+0000 7f39f5ffb640 3 client.14570.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.612+0000 7f39f5ffb640 20 client.14570.objecter dump_active .. 0 homeless 2024-03-25T19:48:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.612+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39fc050000 con 0x7f3a000a98f0 2024-03-25T19:48:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.612+0000 7f39f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.612+0000 7f39f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:18.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.863+0000 7f3a077c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:18.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.863+0000 7f3a077c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:18.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.863+0000 7f3a077c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:18.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.863+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a0013b700 con 0x7f3a000a98f0 2024-03-25T19:48:18.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.964+0000 7f39f5ffb640 1 -- 172.21.15.71:0/834357704 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f39fc052100 con 0x7f3a000a98f0 2024-03-25T19:48:18.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.964+0000 7f39f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:18.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.964+0000 7f39f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:18.964 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:48:18.964 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.966+0000 7f3a077c6640 10 librados: watch_flush enter 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.966+0000 7f3a077c6640 10 librados: watch_flush exit 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.966+0000 7f3a077c6640 20 client.14570.objecter shutdown clearing up homeless session... 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.966+0000 7f3a077c6640 10 client.14570.objecter successfully canceled tick 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.966+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 >> v1:172.21.15.71:6800/3911735624 conn(0x7f39e406dfb0 legacy=0x7f39e4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.966+0000 7f3a077c6640 10 monclient: shutdown 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.966+0000 7f3a077c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.966+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 >> v1:172.21.15.71:6790/0 conn(0x7f3a000a98f0 legacy=0x7f3a001d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:18.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.967+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 shutdown_connections 2024-03-25T19:48:18.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.967+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 >> 172.21.15.71:0/834357704 conn(0x7f3a000b7ea0 msgr2=0x7f3a0014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:18.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.967+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 shutdown_connections 2024-03-25T19:48:18.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.967+0000 7f3a077c6640 1 -- 172.21.15.71:0/834357704 wait complete. 2024-03-25T19:48:18.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:18.967+0000 7f3a077c6640 1 librados: shutdown 2024-03-25T19:48:18.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:22.636 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:05.392 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.686 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:05.442 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.718 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:05.474 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.741 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:05.497 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.746 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:05.465 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.750 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:05.507 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.754 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:05.473 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.796 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:05.515 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.800 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:05.556 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.784 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.884+0000 7f1a2f83e640 10 monclient: get_monmap_and_config 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.884+0000 7f1a2f83e640 10 monclient: build_initial_monmap 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.884+0000 7f1a2f83e640 10 monclient: monmap: 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:48:25.885256+0000 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:48:25.885256+0000 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:25.884 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:25.885 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:25.885 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:48:25.885 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:48:25.885 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.885+0000 7f1a2f83e640 1 Processor -- start 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.885+0000 7f1a2f83e640 1 -- start start 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.885+0000 7f1a2f83e640 10 monclient: init 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 10 monclient(hunting): picked mon.noname-b con 0x7f1a280a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 10 monclient(hunting): picked mon.noname-a con 0x7f1a2813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 10 monclient(hunting): picked mon.noname-c con 0x7f1a281430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a28142520 con 0x7f1a2813f710 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a280b28a0 con 0x7f1a281430c0 2024-03-25T19:48:25.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a280b2a30 con 0x7f1a280a98f0 2024-03-25T19:48:25.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:25.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.886+0000 7f1a2f83e640 10 monclient(hunting): authenticate will time out at 5312.198143s 2024-03-25T19:48:25.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.889+0000 7f1a2dddb640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f1a281430c0 0x7f1a2814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:44538/0 (socket says 172.21.15.71:44538) 2024-03-25T19:48:25.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.889+0000 7f1a2dddb640 1 -- 172.21.15.71:0/2524561960 learned_addr learned my addr 172.21.15.71:0/2524561960 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:25.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.896+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480771529 0 0) 0x7f1a280b28a0 con 0x7f1a281430c0 2024-03-25T19:48:25.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.896+0000 7f1a1ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:25.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.896+0000 7f1a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:25.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.896+0000 7f1a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:25.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.896+0000 7f1a1ffff640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:48:25.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.896+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a14003520 con 0x7f1a281430c0 2024-03-25T19:48:25.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.907+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131278857 0 0) 0x7f1a28142520 con 0x7f1a2813f710 2024-03-25T19:48:25.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.907+0000 7f1a1ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:25.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.907+0000 7f1a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:25.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.907+0000 7f1a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:25.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.907+0000 7f1a1ffff640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:48:25.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.907+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a280b28a0 con 0x7f1a2813f710 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3145179771 0 0) 0x7f1a14003520 con 0x7f1a281430c0 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 >> v1:172.21.15.71:6790/0 conn(0x7f1a280a98f0 legacy=0x7f1a280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 >> v1:172.21.15.71:6789/0 conn(0x7f1a2813f710 legacy=0x7f1a281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 1 monclient: found mon.noname-c 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280b44c0 con 0x7f1a281430c0 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 10 monclient: _finish_auth 0 2024-03-25T19:48:25.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1a20001a10 con 0x7f1a281430c0 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a1ffff640 10 monclient: dump: 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:25.914 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:25.915 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a2f83e640 5 monclient: authenticate success, global_id 4573 2024-03-25T19:48:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.913+0000 7f1a2f83e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:25.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.922+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a20001b70 con 0x7f1a281430c0 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.922+0000 7f1a1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.922+0000 7f1a1ffff640 1 -- 172.21.15.71:0/2524561960 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1a20001f80 con 0x7f1a281430c0 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.922+0000 7f1a1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.922+0000 7f1a1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.922+0000 7f1a1ffff640 10 monclient: dump: 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:25.922 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:25.923 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:25.923 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:25.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.922+0000 7f1a2f83e640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:25.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.922+0000 7f1a2f83e640 4 set_mon_vals no callback set 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.923+0000 7f1a2f83e640 10 monclient: shutdown 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.923+0000 7f1a2f83e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.923+0000 7f1a2f83e640 1 -- 172.21.15.71:0/2524561960 >> v1:172.21.15.193:6789/0 conn(0x7f1a281430c0 legacy=0x7f1a2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.923+0000 7f1a2f83e640 1 -- 172.21.15.71:0/2524561960 shutdown_connections 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.923+0000 7f1a2f83e640 1 -- 172.21.15.71:0/2524561960 >> 172.21.15.71:0/2524561960 conn(0x7f1a280b7ea0 msgr2=0x7f1a280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.923+0000 7f1a2f83e640 1 -- 172.21.15.71:0/2524561960 shutdown_connections 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.923+0000 7f1a2f83e640 1 -- 172.21.15.71:0/2524561960 wait complete. 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 10 monclient: build_initial_monmap 2024-03-25T19:48:25.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 10 monclient: monmap: 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 1 librados: starting msgr at 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 1 librados: starting objecter 2024-03-25T19:48:25.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 1 Processor -- start 2024-03-25T19:48:25.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 1 -- start start 2024-03-25T19:48:25.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 1 librados: setting wanted keys 2024-03-25T19:48:25.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 1 librados: calling monclient init 2024-03-25T19:48:25.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.924+0000 7f1a2f83e640 10 monclient: init 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 10 monclient(hunting): picked mon.noname-a con 0x7f1a280a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2813f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 10 monclient(hunting): picked mon.noname-b con 0x7f1a281430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a28135230 con 0x7f1a280a98f0 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a28135da0 con 0x7f1a281430c0 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a28135f30 con 0x7f1a2813f710 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:25.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.925+0000 7f1a2f83e640 10 monclient(hunting): authenticate will time out at 5312.237074s 2024-03-25T19:48:25.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.935+0000 7f1a2dddb640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f1a281430c0 0x7f1a281db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:44558/0 (socket says 172.21.15.71:44558) 2024-03-25T19:48:25.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.935+0000 7f1a2dddb640 1 -- 172.21.15.71:0/3979383308 learned_addr learned my addr 172.21.15.71:0/3979383308 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:25.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.936+0000 7f1a1dffb640 10 client.?.objecter ms_handle_connect 0x7f1a281430c0 2024-03-25T19:48:25.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.936+0000 7f1a1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:25.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.940+0000 7f1a1dffb640 10 client.?.objecter ms_handle_connect 0x7f1a2813f710 2024-03-25T19:48:25.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.940+0000 7f1a1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:25.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.940+0000 7f1a1dffb640 10 client.?.objecter ms_handle_connect 0x7f1a280a98f0 2024-03-25T19:48:25.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.940+0000 7f1a1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:25.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.946+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244634940 0 0) 0x7f1a28135f30 con 0x7f1a2813f710 2024-03-25T19:48:25.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.946+0000 7f1a1dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:25.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.946+0000 7f1a1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:25.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.946+0000 7f1a1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:25.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.946+0000 7f1a1dffb640 10 monclient(hunting): my global_id is 14576 2024-03-25T19:48:25.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.946+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a0c004eb0 con 0x7f1a2813f710 2024-03-25T19:48:25.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.948+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902837298 0 0) 0x7f1a28135230 con 0x7f1a280a98f0 2024-03-25T19:48:25.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.948+0000 7f1a1dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:25.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.948+0000 7f1a1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:25.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.948+0000 7f1a1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:25.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.948+0000 7f1a1dffb640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:48:25.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.948+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a28135f30 con 0x7f1a280a98f0 2024-03-25T19:48:25.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.958+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2481224479 0 0) 0x7f1a0c004eb0 con 0x7f1a2813f710 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.959+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a28135230 con 0x7f1a2813f710 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.959+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1a18001930 con 0x7f1a2813f710 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.959+0000 7f1a1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.959+0000 7f1a1dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.959+0000 7f1a1dffb640 10 monclient(hunting): dump: 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:25.959 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.959+0000 7f1a1dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.959+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053720243 0 0) 0x7f1a28135da0 con 0x7f1a281430c0 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.960+0000 7f1a1dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.960+0000 7f1a1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.960+0000 7f1a1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.960+0000 7f1a1dffb640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:48:25.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.960+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a0c004eb0 con 0x7f1a281430c0 2024-03-25T19:48:25.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1151191481 0 0) 0x7f1a28135230 con 0x7f1a2813f710 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 >> v1:172.21.15.193:6789/0 conn(0x7f1a281430c0 legacy=0x7f1a281db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 >> v1:172.21.15.71:6789/0 conn(0x7f1a280a98f0 legacy=0x7f1a281d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 1 monclient: found mon.c 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a281360c0 con 0x7f1a2813f710 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 5 monclient: authenticate success, global_id 14576 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 10 monclient: _renew_subs 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:25.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a281d5000 con 0x7f1a2813f710 2024-03-25T19:48:25.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 10 client.14576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:25.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 10 monclient: _renew_subs 2024-03-25T19:48:25.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:25.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a281d5350 con 0x7f1a2813f710 2024-03-25T19:48:25.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.974+0000 7f1a2f83e640 1 librados: init done 2024-03-25T19:48:25.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.975+0000 7f1a2f83e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:25.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.975+0000 7f1a2f83e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:25.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.975+0000 7f1a2f83e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:25.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:25.975+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a28135090 con 0x7f1a2813f710 2024-03-25T19:48:26.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.285+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a18003610 con 0x7f1a2813f710 2024-03-25T19:48:26.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.285+0000 7f1a1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:26.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.285+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1a18003880 con 0x7f1a2813f710 2024-03-25T19:48:26.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.285+0000 7f1a1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:26.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.285+0000 7f1a1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.285+0000 7f1a1dffb640 10 monclient: dump: 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:26.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.285+0000 7f1a1f7fe640 4 set_mon_vals no callback set 2024-03-25T19:48:26.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.286+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f1a18003aa0 con 0x7f1a2813f710 2024-03-25T19:48:26.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.286+0000 7f1a2d5da640 10 monclient: get_auth_request con 0x7f1a0c06de00 auth_method 0 2024-03-25T19:48:26.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.287+0000 7f1a1dffb640 10 client.14576.objecter ms_handle_connect 0x7f1a0c06de00 2024-03-25T19:48:26.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.293+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f1a18053460 con 0x7f1a2813f710 2024-03-25T19:48:26.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.293+0000 7f1a1dffb640 10 client.14576.objecter ms_dispatch 0x7f1a28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:26.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.293+0000 7f1a1dffb640 3 client.14576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:26.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.293+0000 7f1a1dffb640 3 client.14576.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:26.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.293+0000 7f1a1dffb640 20 client.14576.objecter dump_active .. 0 homeless 2024-03-25T19:48:26.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.293+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a1804ef70 con 0x7f1a2813f710 2024-03-25T19:48:26.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.293+0000 7f1a1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:26.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.293+0000 7f1a1dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:26.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.547+0000 7f1a2f83e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:26.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.547+0000 7f1a2f83e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:26.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.547+0000 7f1a2f83e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:26.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.547+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a281d44e0 con 0x7f1a2813f710 2024-03-25T19:48:26.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.570+0000 7f1a1dffb640 1 -- 172.21.15.71:0/3979383308 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f1a18050ea0 con 0x7f1a2813f710 2024-03-25T19:48:26.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.570+0000 7f1a1dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:26.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.570+0000 7f1a1dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:26.570 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:48:26.570 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.572+0000 7f1a2f83e640 10 librados: watch_flush enter 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.572+0000 7f1a2f83e640 10 librados: watch_flush exit 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.572+0000 7f1a2f83e640 20 client.14576.objecter shutdown clearing up homeless session... 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.572+0000 7f1a2f83e640 10 client.14576.objecter successfully canceled tick 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 >> v1:172.21.15.71:6800/3911735624 conn(0x7f1a0c06de00 legacy=0x7f1a0c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 10 monclient: shutdown 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 >> v1:172.21.15.71:6790/0 conn(0x7f1a2813f710 legacy=0x7f1a281dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 shutdown_connections 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 >> 172.21.15.71:0/3979383308 conn(0x7f1a280b7ea0 msgr2=0x7f1a2814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 shutdown_connections 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 1 -- 172.21.15.71:0/3979383308 wait complete. 2024-03-25T19:48:26.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:26.573+0000 7f1a2f83e640 1 librados: shutdown 2024-03-25T19:48:26.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:33.391 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:33.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.506+0000 7f95f5255640 10 monclient: get_monmap_and_config 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.506+0000 7f95f5255640 10 monclient: build_initial_monmap 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.507+0000 7f95f5255640 10 monclient: monmap: 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:48:33.507890+0000 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:48:33.507890+0000 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:48:33.507 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.507+0000 7f95f5255640 1 Processor -- start 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.507+0000 7f95f5255640 1 -- start start 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.507+0000 7f95f5255640 10 monclient: init 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 10 monclient(hunting): picked mon.noname-c con 0x7f95f00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 10 monclient(hunting): picked mon.noname-b con 0x7f95f013f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 10 monclient(hunting): picked mon.noname-a con 0x7f95f01430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:33.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f0142520 con 0x7f95f01430c0 2024-03-25T19:48:33.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f00b28a0 con 0x7f95f00a98f0 2024-03-25T19:48:33.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f00b2a30 con 0x7f95f013f710 2024-03-25T19:48:33.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:33.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.508+0000 7f95f5255640 10 monclient(hunting): authenticate will time out at 5319.820395s 2024-03-25T19:48:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.518+0000 7f95eed9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f95f00a98f0 0x7f95f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:48810/0 (socket says 172.21.15.71:48810) 2024-03-25T19:48:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.518+0000 7f95eed9d640 1 -- 172.21.15.71:0/642010725 learned_addr learned my addr 172.21.15.71:0/642010725 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:33.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.530+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044421230 0 0) 0x7f95f00b28a0 con 0x7f95f00a98f0 2024-03-25T19:48:33.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.530+0000 7f95ee59c640 10 monclient(hunting): _negotiate 2024-03-25T19:48:33.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.530+0000 7f95ee59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:33.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.530+0000 7f95ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:33.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.530+0000 7f95ee59c640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:48:33.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.530+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95dc003520 con 0x7f95f00a98f0 2024-03-25T19:48:33.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.533+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987666041 0 0) 0x7f95f0142520 con 0x7f95f01430c0 2024-03-25T19:48:33.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.533+0000 7f95ee59c640 10 monclient(hunting): _negotiate 2024-03-25T19:48:33.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.533+0000 7f95ee59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:33.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.533+0000 7f95ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:33.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.533+0000 7f95ee59c640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:48:33.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.533+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f00b28a0 con 0x7f95f01430c0 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3205965159 0 0) 0x7f95dc003520 con 0x7f95f00a98f0 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 >> v1:172.21.15.71:6790/0 conn(0x7f95f013f710 legacy=0x7f95f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 >> v1:172.21.15.71:6789/0 conn(0x7f95f01430c0 legacy=0x7f95f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 10 monclient: _finish_hunting 0 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 1 monclient: found mon.noname-c 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f00b44c0 con 0x7f95f00a98f0 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 10 monclient: _finish_auth 0 2024-03-25T19:48:33.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 10 monclient: _check_auth_tickets 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.535+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f95e0001690 con 0x7f95f00a98f0 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.536+0000 7f95f5255640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.536+0000 7f95f5255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.536+0000 7f95ee59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.536+0000 7f95ee59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.536+0000 7f95ee59c640 10 monclient: dump: 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:33.537 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:33.538 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:33.538 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:33.538 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:33.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.536+0000 7f95f5255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:33.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.546+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e00017f0 con 0x7f95f00a98f0 2024-03-25T19:48:33.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.546+0000 7f95ee59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:33.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.546+0000 7f95ee59c640 1 -- 172.21.15.71:0/642010725 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f95e0001c00 con 0x7f95f00a98f0 2024-03-25T19:48:33.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.546+0000 7f95ee59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.546+0000 7f95ee59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.546+0000 7f95ee59c640 10 monclient: dump: 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.547+0000 7f95f5255640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:33.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.547+0000 7f95f5255640 4 set_mon_vals no callback set 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.547+0000 7f95f5255640 10 monclient: shutdown 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.547+0000 7f95f5255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.547+0000 7f95f5255640 1 -- 172.21.15.71:0/642010725 >> v1:172.21.15.193:6789/0 conn(0x7f95f00a98f0 legacy=0x7f95f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.548+0000 7f95f5255640 1 -- 172.21.15.71:0/642010725 shutdown_connections 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.548+0000 7f95f5255640 1 -- 172.21.15.71:0/642010725 >> 172.21.15.71:0/642010725 conn(0x7f95f00b7ea0 msgr2=0x7f95f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.548+0000 7f95f5255640 1 -- 172.21.15.71:0/642010725 shutdown_connections 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.548+0000 7f95f5255640 1 -- 172.21.15.71:0/642010725 wait complete. 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.548+0000 7f95f5255640 10 monclient: build_initial_monmap 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.548+0000 7f95f5255640 10 monclient: monmap: 2024-03-25T19:48:33.549 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 1 librados: starting msgr at 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 1 librados: starting objecter 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 1 Processor -- start 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 1 -- start start 2024-03-25T19:48:33.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 1 librados: setting wanted keys 2024-03-25T19:48:33.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 1 librados: calling monclient init 2024-03-25T19:48:33.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 10 monclient: init 2024-03-25T19:48:33.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:33.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:33.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.549+0000 7f95f5255640 10 monclient(hunting): picked mon.noname-c con 0x7f95f00a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:33.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.550+0000 7f95f5255640 10 monclient(hunting): picked mon.noname-a con 0x7f95f013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:33.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.550+0000 7f95f5255640 10 monclient(hunting): picked mon.noname-b con 0x7f95f01430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:33.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.550+0000 7f95f5255640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f0138250 con 0x7f95f013f710 2024-03-25T19:48:33.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.550+0000 7f95f5255640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f0137d50 con 0x7f95f01430c0 2024-03-25T19:48:33.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.550+0000 7f95f5255640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f0137ee0 con 0x7f95f00a98f0 2024-03-25T19:48:33.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.550+0000 7f95f5255640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:33.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.550+0000 7f95f5255640 10 monclient(hunting): authenticate will time out at 5319.862023s 2024-03-25T19:48:33.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.558+0000 7f95e7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f95f013f710 0x7f95f01d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51446/0 (socket says 172.21.15.71:51446) 2024-03-25T19:48:33.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.558+0000 7f95e7fff640 1 -- 172.21.15.71:0/965242175 learned_addr learned my addr 172.21.15.71:0/965242175 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:33.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.561+0000 7f95e77fe640 10 client.?.objecter ms_handle_connect 0x7f95f01430c0 2024-03-25T19:48:33.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.561+0000 7f95e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:33.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.565+0000 7f95e77fe640 10 client.?.objecter ms_handle_connect 0x7f95f013f710 2024-03-25T19:48:33.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.565+0000 7f95e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:33.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.565+0000 7f95e77fe640 10 client.?.objecter ms_handle_connect 0x7f95f00a98f0 2024-03-25T19:48:33.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.565+0000 7f95e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:33.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.572+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168004892 0 0) 0x7f95f0138250 con 0x7f95f013f710 2024-03-25T19:48:33.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.572+0000 7f95e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:33.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.572+0000 7f95e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:33.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.572+0000 7f95e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:33.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.572+0000 7f95e77fe640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:48:33.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.572+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d0004eb0 con 0x7f95f013f710 2024-03-25T19:48:33.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.577+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056035648 0 0) 0x7f95f0137ee0 con 0x7f95f00a98f0 2024-03-25T19:48:33.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.577+0000 7f95e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.577+0000 7f95e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.577+0000 7f95e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.577+0000 7f95e77fe640 10 monclient(hunting): my global_id is 14579 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.577+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f0138250 con 0x7f95f00a98f0 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.578+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841757015 0 0) 0x7f95f0137d50 con 0x7f95f01430c0 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.578+0000 7f95e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.578+0000 7f95e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.578+0000 7f95e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.578+0000 7f95e77fe640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:48:33.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.578+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f0137ee0 con 0x7f95f01430c0 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.587+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3275451043 0 0) 0x7f95f0137ee0 con 0x7f95f01430c0 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.587+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95f0137d50 con 0x7f95f01430c0 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.587+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f95e8001bd0 con 0x7f95f01430c0 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.587+0000 7f95e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.587+0000 7f95e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.587+0000 7f95e77fe640 10 monclient(hunting): dump: 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.587 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:33.588 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:33.588 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:33.588 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:33.588 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:33.588 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:33.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.587+0000 7f95e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.590+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3937398346 0 0) 0x7f95d0004eb0 con 0x7f95f013f710 2024-03-25T19:48:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.590+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95f0137ee0 con 0x7f95f013f710 2024-03-25T19:48:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.590+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f95d8001a30 con 0x7f95f013f710 2024-03-25T19:48:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.590+0000 7f95e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.590+0000 7f95e77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.590+0000 7f95e77fe640 10 monclient(hunting): dump: 2024-03-25T19:48:33.590 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:33.590 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:33.591 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.591 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.591 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:33.591 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:33.591 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:33.591 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:33.591 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:33.591 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.598+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637566936 0 0) 0x7f95f0138250 con 0x7f95f00a98f0 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.598+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95d0004eb0 con 0x7f95f00a98f0 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.598+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f95e0001b00 con 0x7f95f00a98f0 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.598+0000 7f95e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.598+0000 7f95e77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.598+0000 7f95e77fe640 10 monclient(hunting): dump: 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:33.598 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:33.599 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:33.599 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:33.599 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:33.599 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3837949775 0 0) 0x7f95f0137d50 con 0x7f95f01430c0 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 >> v1:172.21.15.71:6790/0 conn(0x7f95f00a98f0 legacy=0x7f95f0137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 >> v1:172.21.15.71:6789/0 conn(0x7f95f013f710 legacy=0x7f95f01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 1 monclient: found mon.b 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f00b7730 con 0x7f95f01430c0 2024-03-25T19:48:33.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 5 monclient: authenticate success, global_id 4582 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 10 monclient: _renew_subs 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f00b7920 con 0x7f95f01430c0 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 10 monclient: _renew_subs 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f01e33d0 con 0x7f95f01430c0 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.599+0000 7f95f5255640 1 librados: init done 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.600+0000 7f95f5255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.600+0000 7f95f5255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:33.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.600+0000 7f95f5255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:33.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.600+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f0139770 con 0x7f95f01430c0 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.883+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e8001fe0 con 0x7f95f01430c0 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.883+0000 7f95e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.883+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f95e8003970 con 0x7f95f01430c0 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.883+0000 7f95e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.883+0000 7f95e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.883+0000 7f95e77fe640 10 monclient: dump: 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:33.883 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.884 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:33.884 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:33.884 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:33.884 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:33.884 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:33.884 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:33.884 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:33.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.883+0000 7f95edd9b640 4 set_mon_vals no callback set 2024-03-25T19:48:33.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.883+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f95e8019940 con 0x7f95f01430c0 2024-03-25T19:48:33.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.884+0000 7f95eed9d640 10 monclient: get_auth_request con 0x7f95d006e0a0 auth_method 0 2024-03-25T19:48:33.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.884+0000 7f95e77fe640 10 client.4582.objecter ms_handle_connect 0x7f95d006e0a0 2024-03-25T19:48:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.888+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f95e80012e0 con 0x7f95f01430c0 2024-03-25T19:48:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.888+0000 7f95e77fe640 10 client.4582.objecter ms_dispatch 0x7f95f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.888+0000 7f95e77fe640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.888+0000 7f95e77fe640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:33.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.888+0000 7f95e77fe640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T19:48:33.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.889+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95e804ef50 con 0x7f95f01430c0 2024-03-25T19:48:33.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.889+0000 7f95e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:33.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:33.889+0000 7f95e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:34.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.126+0000 7f95f5255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:34.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.126+0000 7f95f5255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:34.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.126+0000 7f95f5255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:34.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.126+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95f0138400 con 0x7f95f01430c0 2024-03-25T19:48:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.153+0000 7f95e77fe640 1 -- 172.21.15.71:0/965242175 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f95f0138400 con 0x7f95f01430c0 2024-03-25T19:48:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.153+0000 7f95e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.153+0000 7f95e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:34.154 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:48:34.154 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:34.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 10 librados: watch_flush enter 2024-03-25T19:48:34.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 10 librados: watch_flush exit 2024-03-25T19:48:34.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T19:48:34.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 10 client.4582.objecter successfully canceled tick 2024-03-25T19:48:34.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 >> v1:172.21.15.71:6800/3911735624 conn(0x7f95d006e0a0 legacy=0x7f95d0070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:34.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 10 monclient: shutdown 2024-03-25T19:48:34.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:34.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 >> v1:172.21.15.193:6789/0 conn(0x7f95f01430c0 legacy=0x7f95f01d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:34.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 shutdown_connections 2024-03-25T19:48:34.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 >> 172.21.15.71:0/965242175 conn(0x7f95f00b7ea0 msgr2=0x7f95f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:34.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 shutdown_connections 2024-03-25T19:48:34.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 1 -- 172.21.15.71:0/965242175 wait complete. 2024-03-25T19:48:34.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:34.156+0000 7f95f5255640 1 librados: shutdown 2024-03-25T19:48:34.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:40.977 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.096+0000 7fd28f97b640 10 monclient: get_monmap_and_config 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.096+0000 7fd28f97b640 10 monclient: build_initial_monmap 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.096+0000 7fd28f97b640 10 monclient: monmap: 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:48:41.098399+0000 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:48:41.098399+0000 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:41.097 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:41.098 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:41.098 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:48:41.098 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:48:41.098 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.097+0000 7fd28f97b640 1 Processor -- start 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.097+0000 7fd28f97b640 1 -- start start 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.097+0000 7fd28f97b640 10 monclient: init 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 10 monclient(hunting): picked mon.noname-a con 0x7fd2880b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 10 monclient(hunting): picked mon.noname-b con 0x7fd2880b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 10 monclient(hunting): picked mon.noname-c con 0x7fd28814be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2880af940 con 0x7fd2880b4d60 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2880b0a70 con 0x7fd28814be90 2024-03-25T19:48:41.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2880b0d50 con 0x7fd2880b1730 2024-03-25T19:48:41.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:41.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.098+0000 7fd28f97b640 10 monclient(hunting): authenticate will time out at 5327.411392s 2024-03-25T19:48:41.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.111+0000 7fd28cf16640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd2880b1730 0x7fd2880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52434/0 (socket says 172.21.15.71:52434) 2024-03-25T19:48:41.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.111+0000 7fd28cf16640 1 -- 172.21.15.71:0/65760078 learned_addr learned my addr 172.21.15.71:0/65760078 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:41.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.118+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127125546 0 0) 0x7fd2880b0d50 con 0x7fd2880b1730 2024-03-25T19:48:41.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.118+0000 7fd27ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:41.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.118+0000 7fd27ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:41.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.118+0000 7fd27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:41.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.118+0000 7fd27ffff640 10 monclient(hunting): my global_id is 14582 2024-03-25T19:48:41.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.118+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd274003520 con 0x7fd2880b1730 2024-03-25T19:48:41.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.123+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944443325 0 0) 0x7fd2880b0a70 con 0x7fd28814be90 2024-03-25T19:48:41.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.123+0000 7fd27ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:41.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.123+0000 7fd27ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:41.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.123+0000 7fd27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:41.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.123+0000 7fd27ffff640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:48:41.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.124+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2880b0d50 con 0x7fd28814be90 2024-03-25T19:48:41.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294657946 0 0) 0x7fd274003520 con 0x7fd2880b1730 2024-03-25T19:48:41.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 >> v1:172.21.15.193:6789/0 conn(0x7fd28814be90 legacy=0x7fd28814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:41.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 >> v1:172.21.15.71:6789/0 conn(0x7fd2880b4d60 legacy=0x7fd28813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:41.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:41.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 1 monclient: found mon.noname-b 2024-03-25T19:48:41.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:41.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2880b1030 con 0x7fd2880b1730 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 10 monclient: _finish_auth 0 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd278001690 con 0x7fd2880b1730 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd27ffff640 10 monclient: dump: 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:41.142 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:41.143 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:41.143 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:41.143 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:41.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd28f97b640 5 monclient: authenticate success, global_id 14582 2024-03-25T19:48:41.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.140+0000 7fd28f97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:41.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2780017f0 con 0x7fd2880b1730 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd27ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd27ffff640 1 -- 172.21.15.71:0/65760078 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd278001c00 con 0x7fd2880b1730 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd27ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd27ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd27ffff640 10 monclient: dump: 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:41.167 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:41.168 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:41.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd28f97b640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:41.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd28f97b640 4 set_mon_vals no callback set 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.166+0000 7fd28f97b640 10 monclient: shutdown 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.167+0000 7fd28f97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.167+0000 7fd28f97b640 1 -- 172.21.15.71:0/65760078 >> v1:172.21.15.71:6790/0 conn(0x7fd2880b1730 legacy=0x7fd2880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.167+0000 7fd28f97b640 1 -- 172.21.15.71:0/65760078 shutdown_connections 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.167+0000 7fd28f97b640 1 -- 172.21.15.71:0/65760078 >> 172.21.15.71:0/65760078 conn(0x7fd2880617a0 msgr2=0x7fd288061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.167+0000 7fd28f97b640 1 -- 172.21.15.71:0/65760078 shutdown_connections 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.167+0000 7fd28f97b640 1 -- 172.21.15.71:0/65760078 wait complete. 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.167+0000 7fd28f97b640 10 monclient: build_initial_monmap 2024-03-25T19:48:41.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.167+0000 7fd28f97b640 10 monclient: monmap: 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.168+0000 7fd28f97b640 1 librados: starting msgr at 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.168+0000 7fd28f97b640 1 librados: starting objecter 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.168+0000 7fd28f97b640 1 Processor -- start 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.168+0000 7fd28f97b640 1 -- start start 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.168+0000 7fd28f97b640 1 librados: setting wanted keys 2024-03-25T19:48:41.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.168+0000 7fd28f97b640 1 librados: calling monclient init 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.168+0000 7fd28f97b640 10 monclient: init 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 10 monclient(hunting): picked mon.noname-b con 0x7fd2880b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 10 monclient(hunting): picked mon.noname-c con 0x7fd2880b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 10 monclient(hunting): picked mon.noname-a con 0x7fd28814be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd288141bf0 con 0x7fd28814be90 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd288141ed0 con 0x7fd2880b1730 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2881421b0 con 0x7fd2880b4d60 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:41.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.169+0000 7fd28f97b640 10 monclient(hunting): authenticate will time out at 5327.482352s 2024-03-25T19:48:41.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.177+0000 7fd28cf16640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd2880b4d60 0x7fd288133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52498/0 (socket says 172.21.15.71:52498) 2024-03-25T19:48:41.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.177+0000 7fd28cf16640 1 -- 172.21.15.71:0/348095679 learned_addr learned my addr 172.21.15.71:0/348095679 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:41.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.181+0000 7fd27dffb640 10 client.?.objecter ms_handle_connect 0x7fd2880b4d60 2024-03-25T19:48:41.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.182+0000 7fd27dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:41.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.186+0000 7fd27dffb640 10 client.?.objecter ms_handle_connect 0x7fd28814be90 2024-03-25T19:48:41.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.186+0000 7fd27dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:41.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.190+0000 7fd27dffb640 10 client.?.objecter ms_handle_connect 0x7fd2880b1730 2024-03-25T19:48:41.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.190+0000 7fd27dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:41.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.192+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992561973 0 0) 0x7fd2881421b0 con 0x7fd2880b4d60 2024-03-25T19:48:41.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.192+0000 7fd27dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:41.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.192+0000 7fd27dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:41.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.192+0000 7fd27dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:41.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.192+0000 7fd27dffb640 10 monclient(hunting): my global_id is 14585 2024-03-25T19:48:41.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.192+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd26c004eb0 con 0x7fd2880b4d60 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.201+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2547008227 0 0) 0x7fd26c004eb0 con 0x7fd2880b4d60 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.201+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2881421b0 con 0x7fd2880b4d60 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.201+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd278001c00 con 0x7fd2880b4d60 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.201+0000 7fd27dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.201+0000 7fd27dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.201+0000 7fd27dffb640 10 monclient(hunting): dump: 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:41.202 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:41.203 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:41.203 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:41.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.201+0000 7fd27dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:41.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.203+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716307773 0 0) 0x7fd288141bf0 con 0x7fd28814be90 2024-03-25T19:48:41.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.203+0000 7fd27dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:41.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.203+0000 7fd27dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:41.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.203+0000 7fd27dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:41.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.203+0000 7fd27dffb640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:48:41.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.203+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd26c004eb0 con 0x7fd28814be90 2024-03-25T19:48:41.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3185647499 0 0) 0x7fd2881421b0 con 0x7fd2880b4d60 2024-03-25T19:48:41.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 >> v1:172.21.15.193:6789/0 conn(0x7fd2880b1730 legacy=0x7fd2881425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:41.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 >> v1:172.21.15.71:6789/0 conn(0x7fd28814be90 legacy=0x7fd2881341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:41.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:41.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 1 monclient: found mon.c 2024-03-25T19:48:41.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:41.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:41.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd288140f00 con 0x7fd2880b4d60 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd27dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 5 monclient: authenticate success, global_id 14585 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 10 monclient: _renew_subs 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2881406f0 con 0x7fd2880b4d60 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 10 client.14585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 10 monclient: _renew_subs 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd288140380 con 0x7fd2880b4d60 2024-03-25T19:48:41.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.213+0000 7fd28f97b640 1 librados: init done 2024-03-25T19:48:41.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.214+0000 7fd28f97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:41.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.214+0000 7fd28f97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:41.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.214+0000 7fd28f97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:41.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.214+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2881401f0 con 0x7fd2880b4d60 2024-03-25T19:48:41.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.426+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd278004a90 con 0x7fd2880b4d60 2024-03-25T19:48:41.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.426+0000 7fd27dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:41.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.426+0000 7fd27f7fe640 4 set_mon_vals no callback set 2024-03-25T19:48:41.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.515+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd278004e60 con 0x7fd2880b4d60 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.515+0000 7fd27dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.515+0000 7fd27dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.515+0000 7fd27dffb640 10 monclient: dump: 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:41.517 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.516+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fd278005040 con 0x7fd2880b4d60 2024-03-25T19:48:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.517+0000 7fd28d717640 10 monclient: get_auth_request con 0x7fd26c06dd90 auth_method 0 2024-03-25T19:48:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.517+0000 7fd27dffb640 10 client.14585.objecter ms_handle_connect 0x7fd26c06dd90 2024-03-25T19:48:41.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.521+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fd2780549a0 con 0x7fd2880b4d60 2024-03-25T19:48:41.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.521+0000 7fd27dffb640 10 client.14585.objecter ms_dispatch 0x7fd288000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:41.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.521+0000 7fd27dffb640 3 client.14585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:41.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.521+0000 7fd27dffb640 3 client.14585.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:41.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.522+0000 7fd27dffb640 20 client.14585.objecter dump_active .. 0 homeless 2024-03-25T19:48:41.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.522+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2780504b0 con 0x7fd2880b4d60 2024-03-25T19:48:41.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.522+0000 7fd27dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:41.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.522+0000 7fd27dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:41.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.775+0000 7fd28f97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:41.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.775+0000 7fd28f97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:41.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.775+0000 7fd28f97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:41.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.775+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2880627b0 con 0x7fd2880b4d60 2024-03-25T19:48:41.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.896+0000 7fd27dffb640 1 -- 172.21.15.71:0/348095679 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fd2780523e0 con 0x7fd2880b4d60 2024-03-25T19:48:41.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.896+0000 7fd27dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:41.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.896+0000 7fd27dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:41.898 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:48:41.898 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:41.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 10 librados: watch_flush enter 2024-03-25T19:48:41.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 10 librados: watch_flush exit 2024-03-25T19:48:41.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 20 client.14585.objecter shutdown clearing up homeless session... 2024-03-25T19:48:41.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 10 client.14585.objecter successfully canceled tick 2024-03-25T19:48:41.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 >> v1:172.21.15.71:6800/3911735624 conn(0x7fd26c06dd90 legacy=0x7fd26c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:41.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 10 monclient: shutdown 2024-03-25T19:48:41.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:41.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 >> v1:172.21.15.71:6790/0 conn(0x7fd2880b4d60 legacy=0x7fd288133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:41.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 shutdown_connections 2024-03-25T19:48:41.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 >> 172.21.15.71:0/348095679 conn(0x7fd2880617a0 msgr2=0x7fd28813c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:41.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 shutdown_connections 2024-03-25T19:48:41.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 1 -- 172.21.15.71:0/348095679 wait complete. 2024-03-25T19:48:41.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:41.899+0000 7fd28f97b640 1 librados: shutdown 2024-03-25T19:48:41.912 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:42.854 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:25.610 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.903 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:25.659 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.944 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:25.700 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.959 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:25.715 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.968 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:25.724 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.990 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:25.709 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.996 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:25.715 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.002 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:25.759 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.018 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:25.737 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:48.719 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.816+0000 7fef22a2a640 10 monclient: get_monmap_and_config 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.816+0000 7fef22a2a640 10 monclient: build_initial_monmap 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.816+0000 7fef22a2a640 10 monclient: monmap: 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:48:48.817198+0000 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:48:48.817198+0000 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:48.816 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:48.817 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:48:48.817 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:48:48.817 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.817+0000 7fef22a2a640 1 Processor -- start 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.817+0000 7fef22a2a640 1 -- start start 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.817+0000 7fef22a2a640 10 monclient: init 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 10 monclient(hunting): picked mon.noname-a con 0x7fef1c0a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 10 monclient(hunting): picked mon.noname-b con 0x7fef1c13f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 10 monclient(hunting): picked mon.noname-c con 0x7fef1c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c142520 con 0x7fef1c0a98f0 2024-03-25T19:48:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0b28a0 con 0x7fef1c1430c0 2024-03-25T19:48:48.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0b2a30 con 0x7fef1c13f710 2024-03-25T19:48:48.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:48.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.818+0000 7fef22a2a640 10 monclient(hunting): authenticate will time out at 5335.130027s 2024-03-25T19:48:48.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.827+0000 7fef1bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fef1c0a98f0 0x7fef1c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:40558/0 (socket says 172.21.15.71:40558) 2024-03-25T19:48:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.827+0000 7fef1bfff640 1 -- 172.21.15.71:0/839868411 learned_addr learned my addr 172.21.15.71:0/839868411 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.837+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398291772 0 0) 0x7fef1c142520 con 0x7fef1c0a98f0 2024-03-25T19:48:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.837+0000 7fef1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.837+0000 7fef1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.837+0000 7fef1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:48.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.837+0000 7fef1b7fe640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:48:48.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.837+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef08003520 con 0x7fef1c0a98f0 2024-03-25T19:48:48.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.865+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944166927 0 0) 0x7fef1c0b2a30 con 0x7fef1c13f710 2024-03-25T19:48:48.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.865+0000 7fef1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:48.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.865+0000 7fef1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:48.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.865+0000 7fef1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:48.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.865+0000 7fef1b7fe640 10 monclient(hunting): my global_id is 14588 2024-03-25T19:48:48.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.865+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef1c142520 con 0x7fef1c13f710 2024-03-25T19:48:48.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1086455271 0 0) 0x7fef08003520 con 0x7fef1c0a98f0 2024-03-25T19:48:48.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 >> v1:172.21.15.71:6790/0 conn(0x7fef1c13f710 legacy=0x7fef1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:48.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 >> v1:172.21.15.193:6789/0 conn(0x7fef1c1430c0 legacy=0x7fef1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:48.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:48.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:48:48.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:48.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c0b44c0 con 0x7fef1c0a98f0 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fef0c001690 con 0x7fef1c0a98f0 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.868+0000 7fef1b7fe640 10 monclient: dump: 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:48.870 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.869+0000 7fef22a2a640 5 monclient: authenticate success, global_id 4593 2024-03-25T19:48:48.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.869+0000 7fef22a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:48.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef0c0017f0 con 0x7fef1c0a98f0 2024-03-25T19:48:48.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef1b7fe640 1 -- 172.21.15.71:0/839868411 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fef0c001c00 con 0x7fef1c0a98f0 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef1b7fe640 10 monclient: dump: 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:48.876 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:48.877 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:48.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef22a2a640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:48.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef22a2a640 4 set_mon_vals no callback set 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef22a2a640 10 monclient: shutdown 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef22a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.876+0000 7fef22a2a640 1 -- 172.21.15.71:0/839868411 >> v1:172.21.15.71:6789/0 conn(0x7fef1c0a98f0 legacy=0x7fef1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.877+0000 7fef22a2a640 1 -- 172.21.15.71:0/839868411 shutdown_connections 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.877+0000 7fef22a2a640 1 -- 172.21.15.71:0/839868411 >> 172.21.15.71:0/839868411 conn(0x7fef1c0b7ea0 msgr2=0x7fef1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.877+0000 7fef22a2a640 1 -- 172.21.15.71:0/839868411 shutdown_connections 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.877+0000 7fef22a2a640 1 -- 172.21.15.71:0/839868411 wait complete. 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.877+0000 7fef22a2a640 10 monclient: build_initial_monmap 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.877+0000 7fef22a2a640 10 monclient: monmap: 2024-03-25T19:48:48.879 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.878+0000 7fef22a2a640 1 librados: starting msgr at 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.878+0000 7fef22a2a640 1 librados: starting objecter 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.878+0000 7fef22a2a640 1 Processor -- start 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.878+0000 7fef22a2a640 1 -- start start 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.878+0000 7fef22a2a640 1 librados: setting wanted keys 2024-03-25T19:48:48.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.878+0000 7fef22a2a640 1 librados: calling monclient init 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.878+0000 7fef22a2a640 10 monclient: init 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 10 monclient(hunting): picked mon.noname-a con 0x7fef1c0a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 10 monclient(hunting): picked mon.noname-c con 0x7fef1c13f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 10 monclient(hunting): picked mon.noname-b con 0x7fef1c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c138970 con 0x7fef1c0a98f0 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c1d7810 con 0x7fef1c1430c0 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c139990 con 0x7fef1c13f710 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:48.881 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.879+0000 7fef22a2a640 10 monclient(hunting): authenticate will time out at 5335.191191s 2024-03-25T19:48:48.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.882+0000 7fef13fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fef1c13f710 0x7fef1c1d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35492/0 (socket says 172.21.15.71:35492) 2024-03-25T19:48:48.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.882+0000 7fef13fff640 1 -- 172.21.15.71:0/2841793414 learned_addr learned my addr 172.21.15.71:0/2841793414 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:48.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.883+0000 7fef197fa640 10 client.?.objecter ms_handle_connect 0x7fef1c13f710 2024-03-25T19:48:48.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.883+0000 7fef197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:48.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.890+0000 7fef197fa640 10 client.?.objecter ms_handle_connect 0x7fef1c0a98f0 2024-03-25T19:48:48.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.890+0000 7fef197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:48.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.892+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129440610 0 0) 0x7fef1c139990 con 0x7fef1c13f710 2024-03-25T19:48:48.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.892+0000 7fef197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:48.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.892+0000 7fef197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:48.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.892+0000 7fef197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:48.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.892+0000 7fef197fa640 10 monclient(hunting): my global_id is 14591 2024-03-25T19:48:48.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.892+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feefc004eb0 con 0x7fef1c13f710 2024-03-25T19:48:48.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.893+0000 7fef197fa640 10 client.?.objecter ms_handle_connect 0x7fef1c1430c0 2024-03-25T19:48:48.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.893+0000 7fef197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:48.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.909+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859756851 0 0) 0x7fef1c138970 con 0x7fef1c0a98f0 2024-03-25T19:48:48.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.909+0000 7fef197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:48.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.909+0000 7fef197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:48.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.909+0000 7fef197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:48.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.909+0000 7fef197fa640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:48:48.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.909+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef1c139990 con 0x7fef1c0a98f0 2024-03-25T19:48:48.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.910+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206164103 0 0) 0x7fef1c1d7810 con 0x7fef1c1430c0 2024-03-25T19:48:48.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.910+0000 7fef197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:48.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.910+0000 7fef197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:48.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.910+0000 7fef197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:48.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.910+0000 7fef197fa640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:48:48.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.910+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef1c138970 con 0x7fef1c1430c0 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.926+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272608525 0 0) 0x7feefc004eb0 con 0x7fef1c13f710 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.926+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef1c1d7810 con 0x7fef1c13f710 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.926+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fef04001a10 con 0x7fef1c13f710 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.926+0000 7fef197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.926+0000 7fef197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.926+0000 7fef197fa640 10 monclient(hunting): dump: 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:48.926 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:48.927 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:48.927 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:48.927 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:48.927 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:48.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.926+0000 7fef197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:48.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.928+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418013506 0 0) 0x7fef1c139990 con 0x7fef1c0a98f0 2024-03-25T19:48:48.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.928+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feefc004eb0 con 0x7fef1c0a98f0 2024-03-25T19:48:48.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.928+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fef0c001b00 con 0x7fef1c0a98f0 2024-03-25T19:48:48.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.928+0000 7fef197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:48.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.928+0000 7fef197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:48.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.928+0000 7fef197fa640 10 monclient(hunting): dump: 2024-03-25T19:48:48.928 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:48.928 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:48.929 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.929 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.929 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:48.929 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:48.929 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:48.929 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:48.929 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:48.929 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.930+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 642777030 0 0) 0x7fef1c138970 con 0x7fef1c1430c0 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.930+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef1c139990 con 0x7fef1c1430c0 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.930+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fef14001a90 con 0x7fef1c1430c0 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.930+0000 7fef197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.930+0000 7fef197fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.930+0000 7fef197fa640 10 monclient(hunting): dump: 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:48.930 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.931 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:48.931 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:48.931 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:48.931 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:48.931 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:48.931 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:48.931 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:48.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 224064622 0 0) 0x7fef1c1d7810 con 0x7fef1c13f710 2024-03-25T19:48:48.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 >> v1:172.21.15.193:6789/0 conn(0x7fef1c1430c0 legacy=0x7fef1c1d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:48.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 >> v1:172.21.15.71:6789/0 conn(0x7fef1c0a98f0 legacy=0x7fef1c138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 10 monclient: _finish_hunting 0 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 1 monclient: found mon.c 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c1d8440 con 0x7fef1c13f710 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 10 monclient: _finish_auth 0 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 10 monclient: _check_auth_tickets 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 5 monclient: authenticate success, global_id 14591 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 10 monclient: _renew_subs 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c1d41e0 con 0x7fef1c13f710 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 10 client.14591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:48.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 10 monclient: _renew_subs 2024-03-25T19:48:48.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:48.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c138d80 con 0x7fef1c13f710 2024-03-25T19:48:48.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.943+0000 7fef22a2a640 1 librados: init done 2024-03-25T19:48:48.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.944+0000 7fef22a2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:48.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.944+0000 7fef22a2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:48.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.944+0000 7fef22a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:48.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:48.944+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1c1d79a0 con 0x7fef1c13f710 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.268+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef04001c50 con 0x7fef1c13f710 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.268+0000 7fef197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.268+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fef04003a60 con 0x7fef1c13f710 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.268+0000 7fef197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.268+0000 7fef197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.268+0000 7fef197fa640 10 monclient: dump: 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:49.268 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:49.269 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:49.269 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:49.269 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:49.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.268+0000 7fef1affd640 4 set_mon_vals no callback set 2024-03-25T19:48:49.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.269+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fef04003c80 con 0x7fef1c13f710 2024-03-25T19:48:49.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.269+0000 7fef1bfff640 10 monclient: get_auth_request con 0x7feefc06e0f0 auth_method 0 2024-03-25T19:48:49.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.270+0000 7fef197fa640 10 client.14591.objecter ms_handle_connect 0x7feefc06e0f0 2024-03-25T19:48:49.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.274+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fef040548b0 con 0x7fef1c13f710 2024-03-25T19:48:49.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.274+0000 7fef197fa640 10 client.14591.objecter ms_dispatch 0x7fef1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:49.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.274+0000 7fef197fa640 3 client.14591.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:49.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.274+0000 7fef197fa640 3 client.14591.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:49.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.274+0000 7fef197fa640 20 client.14591.objecter dump_active .. 0 homeless 2024-03-25T19:48:49.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.274+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef040503c0 con 0x7fef1c13f710 2024-03-25T19:48:49.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.274+0000 7fef197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:49.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.274+0000 7fef197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:49.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.523+0000 7fef22a2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:49.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.523+0000 7fef22a2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:49.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.523+0000 7fef22a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:48:49.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.523+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef1c0627b0 con 0x7fef1c13f710 2024-03-25T19:48:49.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.552+0000 7fef197fa640 1 -- 172.21.15.71:0/2841793414 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fef040522f0 con 0x7fef1c13f710 2024-03-25T19:48:49.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.552+0000 7fef197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:49.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.552+0000 7fef197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:49.552 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:48:49.552 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:49.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 10 librados: watch_flush enter 2024-03-25T19:48:49.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 10 librados: watch_flush exit 2024-03-25T19:48:49.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 20 client.14591.objecter shutdown clearing up homeless session... 2024-03-25T19:48:49.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 10 client.14591.objecter successfully canceled tick 2024-03-25T19:48:49.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 >> v1:172.21.15.71:6800/3911735624 conn(0x7feefc06e0f0 legacy=0x7feefc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:49.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 10 monclient: shutdown 2024-03-25T19:48:49.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:49.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 >> v1:172.21.15.71:6790/0 conn(0x7fef1c13f710 legacy=0x7fef1c1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:49.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 shutdown_connections 2024-03-25T19:48:49.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 >> 172.21.15.71:0/2841793414 conn(0x7fef1c0b7ea0 msgr2=0x7fef1c13c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:49.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 shutdown_connections 2024-03-25T19:48:49.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 1 -- 172.21.15.71:0/2841793414 wait complete. 2024-03-25T19:48:49.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:49.554+0000 7fef22a2a640 1 librados: shutdown 2024-03-25T19:48:49.564 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:56.370 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.488+0000 7f175062a640 10 monclient: get_monmap_and_config 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.488+0000 7f175062a640 10 monclient: build_initial_monmap 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.488+0000 7f175062a640 10 monclient: monmap: 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:48:56.489378+0000 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:48:56.489378+0000 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:56.488 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:48:56.489 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:48:56.489 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.489+0000 7f175062a640 1 Processor -- start 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.489+0000 7f175062a640 1 -- start start 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.489+0000 7f175062a640 10 monclient: init 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 10 monclient(hunting): picked mon.noname-b con 0x7f174806ab80 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 10 monclient(hunting): picked mon.noname-a con 0x7f174806af60 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 10 monclient(hunting): picked mon.noname-c con 0x7f17480b00b0 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17480b0500 con 0x7f174806af60 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174813ffa0 con 0x7f17480b00b0 2024-03-25T19:48:56.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17480af520 con 0x7f174806ab80 2024-03-25T19:48:56.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:56.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.490+0000 7f175062a640 10 monclient(hunting): authenticate will time out at 5342.802067s 2024-03-25T19:48:56.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.494+0000 7f174ebc7640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f17480b00b0 0x7f174814dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:45984/0 (socket says 172.21.15.71:45984) 2024-03-25T19:48:56.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.494+0000 7f174ebc7640 1 -- 172.21.15.71:0/3348206814 learned_addr learned my addr 172.21.15.71:0/3348206814 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:56.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.506+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068289288 0 0) 0x7f174813ffa0 con 0x7f17480b00b0 2024-03-25T19:48:56.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.506+0000 7f174d3c4640 10 monclient(hunting): _negotiate 2024-03-25T19:48:56.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.506+0000 7f174d3c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:56.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.506+0000 7f174d3c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:56.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.506+0000 7f174d3c4640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:48:56.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.506+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1734003520 con 0x7f17480b00b0 2024-03-25T19:48:56.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.507+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579749 0 0) 0x7f17480b0500 con 0x7f174806af60 2024-03-25T19:48:56.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.507+0000 7f174d3c4640 10 monclient(hunting): _negotiate 2024-03-25T19:48:56.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.507+0000 7f174d3c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:56.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.507+0000 7f174d3c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:56.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.507+0000 7f174d3c4640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:48:56.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.507+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f174813ffa0 con 0x7f174806af60 2024-03-25T19:48:56.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648924620 0 0) 0x7f1734003520 con 0x7f17480b00b0 2024-03-25T19:48:56.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 >> v1:172.21.15.71:6790/0 conn(0x7f174806ab80 legacy=0x7f17480ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:56.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 >> v1:172.21.15.71:6789/0 conn(0x7f174806af60 legacy=0x7f174813f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:56.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 10 monclient: _finish_hunting 0 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 1 monclient: found mon.noname-c 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1748072890 con 0x7f17480b00b0 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 10 monclient: _finish_auth 0 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 10 monclient: _check_auth_tickets 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1740001a10 con 0x7f17480b00b0 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f174d3c4640 10 monclient: dump: 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:56.513 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f175062a640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:48:56.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.512+0000 7f175062a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:56.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.514+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1740001b70 con 0x7f17480b00b0 2024-03-25T19:48:56.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.514+0000 7f174d3c4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:56.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.514+0000 7f174d3c4640 1 -- 172.21.15.71:0/3348206814 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1740001f80 con 0x7f17480b00b0 2024-03-25T19:48:56.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.514+0000 7f175062a640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:56.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.514+0000 7f175062a640 4 set_mon_vals no callback set 2024-03-25T19:48:56.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.514+0000 7f174d3c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:56.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.514+0000 7f174d3c4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:56.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.514+0000 7f174d3c4640 10 monclient: dump: 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.515+0000 7f175062a640 10 monclient: shutdown 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.515+0000 7f175062a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.515+0000 7f175062a640 1 -- 172.21.15.71:0/3348206814 >> v1:172.21.15.193:6789/0 conn(0x7f17480b00b0 legacy=0x7f174814dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:56.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.515+0000 7f175062a640 1 -- 172.21.15.71:0/3348206814 shutdown_connections 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.515+0000 7f175062a640 1 -- 172.21.15.71:0/3348206814 >> 172.21.15.71:0/3348206814 conn(0x7f17480b4a80 msgr2=0x7f17480b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.515+0000 7f175062a640 1 -- 172.21.15.71:0/3348206814 shutdown_connections 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.515+0000 7f175062a640 1 -- 172.21.15.71:0/3348206814 wait complete. 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 10 monclient: build_initial_monmap 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 10 monclient: monmap: 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:48:56.518 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:48:56.519 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:56.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 1 librados: starting msgr at 2024-03-25T19:48:56.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 1 librados: starting objecter 2024-03-25T19:48:56.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 1 Processor -- start 2024-03-25T19:48:56.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 1 -- start start 2024-03-25T19:48:56.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 1 librados: setting wanted keys 2024-03-25T19:48:56.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 1 librados: calling monclient init 2024-03-25T19:48:56.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.516+0000 7f175062a640 10 monclient: init 2024-03-25T19:48:56.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:56.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:56.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 10 monclient(hunting): picked mon.noname-b con 0x7f174806ab80 addr v1:172.21.15.193:6789/0 2024-03-25T19:48:56.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 10 monclient(hunting): picked mon.noname-c con 0x7f17480b00b0 addr v1:172.21.15.71:6790/0 2024-03-25T19:48:56.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 10 monclient(hunting): picked mon.noname-a con 0x7f1748143990 addr v1:172.21.15.71:6789/0 2024-03-25T19:48:56.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17481d72d0 con 0x7f1748143990 2024-03-25T19:48:56.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17481450b0 con 0x7f174806ab80 2024-03-25T19:48:56.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17481388b0 con 0x7f17480b00b0 2024-03-25T19:48:56.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:56.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.517+0000 7f175062a640 10 monclient(hunting): authenticate will time out at 5342.829282s 2024-03-25T19:48:56.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.522+0000 7f174ebc7640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1748143990 0x7f17481e2bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:55362/0 (socket says 172.21.15.71:55362) 2024-03-25T19:48:56.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.522+0000 7f174ebc7640 1 -- 172.21.15.71:0/3056369837 learned_addr learned my addr 172.21.15.71:0/3056369837 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:48:56.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.527+0000 7f173effd640 10 client.?.objecter ms_handle_connect 0x7f1748143990 2024-03-25T19:48:56.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.527+0000 7f173effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:56.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.532+0000 7f173effd640 10 client.?.objecter ms_handle_connect 0x7f17480b00b0 2024-03-25T19:48:56.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.532+0000 7f173effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:56.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.538+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760722858 0 0) 0x7f17481388b0 con 0x7f17480b00b0 2024-03-25T19:48:56.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.538+0000 7f173effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:56.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.538+0000 7f173effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:56.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.538+0000 7f173effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:56.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.538+0000 7f173effd640 10 monclient(hunting): my global_id is 14597 2024-03-25T19:48:56.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.538+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f172c004eb0 con 0x7f17480b00b0 2024-03-25T19:48:56.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.538+0000 7f173effd640 10 client.?.objecter ms_handle_connect 0x7f174806ab80 2024-03-25T19:48:56.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.538+0000 7f173effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.540+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330151574 0 0) 0x7f17481d72d0 con 0x7f1748143990 2024-03-25T19:48:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.540+0000 7f173effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.540+0000 7f173effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.540+0000 7f173effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.540+0000 7f173effd640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:48:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.540+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17481388b0 con 0x7f1748143990 2024-03-25T19:48:56.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.546+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669573564 0 0) 0x7f17481450b0 con 0x7f174806ab80 2024-03-25T19:48:56.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.546+0000 7f173effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:56.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.546+0000 7f173effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:56.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.546+0000 7f173effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:56.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.546+0000 7f173effd640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:48:56.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.546+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17481d72d0 con 0x7f174806ab80 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.553+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1968592533 0 0) 0x7f172c004eb0 con 0x7f17480b00b0 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.553+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17481450b0 con 0x7f17480b00b0 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.553+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1738001930 con 0x7f17480b00b0 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.553+0000 7f173effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.553+0000 7f173effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.553+0000 7f173effd640 10 monclient(hunting): dump: 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:56.553 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:56.554 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:56.554 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:56.554 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:56.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.553+0000 7f173effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:56.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.560+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3666277756 0 0) 0x7f17481d72d0 con 0x7f174806ab80 2024-03-25T19:48:56.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.560+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f172c004eb0 con 0x7f174806ab80 2024-03-25T19:48:56.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.560+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1744001870 con 0x7f174806ab80 2024-03-25T19:48:56.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.560+0000 7f173effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:56.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.560+0000 7f173effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:56.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.560+0000 7f173effd640 10 monclient(hunting): dump: 2024-03-25T19:48:56.560 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:56.560 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:56.561 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.561 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.561 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:56.561 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:56.561 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:56.561 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:56.561 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:56.561 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:56.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.562+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1617584113 0 0) 0x7f17481388b0 con 0x7f1748143990 2024-03-25T19:48:56.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.562+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17481d72d0 con 0x7f1748143990 2024-03-25T19:48:56.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.562+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1740001e80 con 0x7f1748143990 2024-03-25T19:48:56.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.562+0000 7f173effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:56.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.562+0000 7f173effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:56.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.562+0000 7f173effd640 10 monclient(hunting): dump: 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:56.563 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:56.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1549277783 0 0) 0x7f172c004eb0 con 0x7f174806ab80 2024-03-25T19:48:56.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 >> v1:172.21.15.71:6790/0 conn(0x7f17480b00b0 legacy=0x7f17481dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 >> v1:172.21.15.71:6789/0 conn(0x7f1748143990 legacy=0x7f17481e2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 1 monclient: found mon.b 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1748142e70 con 0x7f174806ab80 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 10 monclient: _finish_auth 0 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f173effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 5 monclient: authenticate success, global_id 4600 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 10 monclient: _renew_subs 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1748144070 con 0x7f174806ab80 2024-03-25T19:48:56.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:56.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 10 monclient: _renew_subs 2024-03-25T19:48:56.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:56.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17481d8010 con 0x7f174806ab80 2024-03-25T19:48:56.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.574+0000 7f175062a640 1 librados: init done 2024-03-25T19:48:56.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.575+0000 7f175062a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:56.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.575+0000 7f175062a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:56.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.575+0000 7f175062a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:56.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.575+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17480627b0 con 0x7f174806ab80 2024-03-25T19:48:56.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.858+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17440035f0 con 0x7f174806ab80 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.858+0000 7f173effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.858+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f1744001d80 con 0x7f174806ab80 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.858+0000 7f173effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.858+0000 7f173effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.858+0000 7f174cbc3640 4 set_mon_vals no callback set 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.858+0000 7f173effd640 10 monclient: dump: 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:56.859 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:48:56.860 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:48:56.860 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:48:56.860 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:48:56.860 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:48:56.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.859+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f17440197f0 con 0x7f174806ab80 2024-03-25T19:48:56.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.860+0000 7f174dbc5640 10 monclient: get_auth_request con 0x7f172c06e110 auth_method 0 2024-03-25T19:48:56.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.860+0000 7f173effd640 10 client.4600.objecter ms_handle_connect 0x7f172c06e110 2024-03-25T19:48:56.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.865+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f1744053590 con 0x7f174806ab80 2024-03-25T19:48:56.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.865+0000 7f173effd640 10 client.4600.objecter ms_dispatch 0x7f1748000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:56.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.865+0000 7f173effd640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:56.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.865+0000 7f173effd640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:56.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.865+0000 7f173effd640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T19:48:56.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.865+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f174404eed0 con 0x7f174806ab80 2024-03-25T19:48:56.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.865+0000 7f173effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:56.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:56.865+0000 7f173effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:57.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.094+0000 7f175062a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:57.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.094+0000 7f175062a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:57.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.094+0000 7f175062a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:48:57.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.094+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17480b3d80 con 0x7f174806ab80 2024-03-25T19:48:57.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.114+0000 7f173effd640 1 -- 172.21.15.71:0/3056369837 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f1744050fd0 con 0x7f174806ab80 2024-03-25T19:48:57.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.114+0000 7f173effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:57.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.114+0000 7f173effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:57.114 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:48:57.114 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.116+0000 7f175062a640 10 librados: watch_flush enter 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.116+0000 7f175062a640 10 librados: watch_flush exit 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.116+0000 7f175062a640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.116+0000 7f175062a640 10 client.4600.objecter successfully canceled tick 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.116+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 >> v1:172.21.15.71:6800/3911735624 conn(0x7f172c06e110 legacy=0x7f172c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.117+0000 7f175062a640 10 monclient: shutdown 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.117+0000 7f175062a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.117+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 >> v1:172.21.15.193:6789/0 conn(0x7f174806ab80 legacy=0x7f17481d7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.117+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 shutdown_connections 2024-03-25T19:48:57.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.117+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 >> 172.21.15.71:0/3056369837 conn(0x7f17480b4a80 msgr2=0x7f174813f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:57.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.117+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 shutdown_connections 2024-03-25T19:48:57.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.117+0000 7f175062a640 1 -- 172.21.15.71:0/3056369837 wait complete. 2024-03-25T19:48:57.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:48:57.117+0000 7f175062a640 1 librados: shutdown 2024-03-25T19:48:57.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:02.818 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:45.574 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.824 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:45.581 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.862 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:08:45.618 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.879 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:45.635 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.886 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:45.642 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.887 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:45.606 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.891 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:45.610 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.916 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:08:45.672 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.917 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:45.636 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:03.934 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:04.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.040+0000 7fca76089640 10 monclient: get_monmap_and_config 2024-03-25T19:49:04.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.040+0000 7fca76089640 10 monclient: build_initial_monmap 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.040+0000 7fca76089640 10 monclient: monmap: 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:49:04.041794+0000 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:49:04.041794+0000 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:49:04.041 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:04.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.041+0000 7fca76089640 1 Processor -- start 2024-03-25T19:49:04.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.041+0000 7fca76089640 1 -- start start 2024-03-25T19:49:04.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.041+0000 7fca76089640 10 monclient: init 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 10 monclient(hunting): picked mon.noname-a con 0x7fca700b10d0 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 10 monclient(hunting): picked mon.noname-c con 0x7fca700b4360 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 10 monclient(hunting): picked mon.noname-b con 0x7fca700a9470 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700a9ab0 con 0x7fca700b10d0 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700b2400 con 0x7fca700b4360 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700aa2a0 con 0x7fca700a9470 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:04.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.042+0000 7fca76089640 10 monclient(hunting): authenticate will time out at 5350.354125s 2024-03-25T19:49:04.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.056+0000 7fca6effd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fca700b4360 0x7fca700b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:50632/0 (socket says 172.21.15.71:50632) 2024-03-25T19:49:04.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.056+0000 7fca6effd640 1 -- 172.21.15.71:0/2986037145 learned_addr learned my addr 172.21.15.71:0/2986037145 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:04.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.083+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900328706 0 0) 0x7fca700b2400 con 0x7fca700b4360 2024-03-25T19:49:04.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.083+0000 7fca6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:04.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.083+0000 7fca6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:04.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.083+0000 7fca6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:04.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.083+0000 7fca6e7fc640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:49:04.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.083+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca58003520 con 0x7fca700b4360 2024-03-25T19:49:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.092+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958821616 0 0) 0x7fca700aa2a0 con 0x7fca700a9470 2024-03-25T19:49:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.092+0000 7fca6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.092+0000 7fca6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.092+0000 7fca6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.092+0000 7fca6e7fc640 10 monclient(hunting): my global_id is 14600 2024-03-25T19:49:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.092+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca700b2400 con 0x7fca700a9470 2024-03-25T19:49:04.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.094+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842487541 0 0) 0x7fca700a9ab0 con 0x7fca700b10d0 2024-03-25T19:49:04.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.094+0000 7fca6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:04.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.094+0000 7fca6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:04.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.094+0000 7fca6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:04.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.094+0000 7fca6e7fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:49:04.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.094+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca700aa2a0 con 0x7fca700b10d0 2024-03-25T19:49:04.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 535917620 0 0) 0x7fca700b2400 con 0x7fca700a9470 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 >> v1:172.21.15.193:6789/0 conn(0x7fca700b4360 legacy=0x7fca700b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 >> v1:172.21.15.71:6789/0 conn(0x7fca700b10d0 legacy=0x7fca700b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700b14b0 con 0x7fca700a9470 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fca680019f0 con 0x7fca700a9470 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca6e7fc640 10 monclient: dump: 2024-03-25T19:49:04.101 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca76089640 5 monclient: authenticate success, global_id 14600 2024-03-25T19:49:04.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.100+0000 7fca76089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:04.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.116+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca68001b50 con 0x7fca700a9470 2024-03-25T19:49:04.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.116+0000 7fca6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:04.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.116+0000 7fca6e7fc640 1 -- 172.21.15.71:0/2986037145 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fca68001f60 con 0x7fca700a9470 2024-03-25T19:49:04.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.116+0000 7fca76089640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:04.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.116+0000 7fca76089640 4 set_mon_vals no callback set 2024-03-25T19:49:04.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.116+0000 7fca6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:04.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.116+0000 7fca6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.116+0000 7fca6e7fc640 10 monclient: dump: 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:04.117 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:04.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.117+0000 7fca76089640 10 monclient: shutdown 2024-03-25T19:49:04.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.117+0000 7fca76089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.117+0000 7fca76089640 1 -- 172.21.15.71:0/2986037145 >> v1:172.21.15.71:6790/0 conn(0x7fca700a9470 legacy=0x7fca7014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.117+0000 7fca76089640 1 -- 172.21.15.71:0/2986037145 shutdown_connections 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.117+0000 7fca76089640 1 -- 172.21.15.71:0/2986037145 >> 172.21.15.71:0/2986037145 conn(0x7fca700617a0 msgr2=0x7fca70061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.117+0000 7fca76089640 1 -- 172.21.15.71:0/2986037145 shutdown_connections 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.117+0000 7fca76089640 1 -- 172.21.15.71:0/2986037145 wait complete. 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.118+0000 7fca76089640 10 monclient: build_initial_monmap 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.118+0000 7fca76089640 10 monclient: monmap: 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:04.119 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:49:04.120 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:49:04.120 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:04.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.119+0000 7fca76089640 1 librados: starting msgr at 2024-03-25T19:49:04.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.119+0000 7fca76089640 1 librados: starting objecter 2024-03-25T19:49:04.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.119+0000 7fca76089640 1 Processor -- start 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.119+0000 7fca76089640 1 -- start start 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.119+0000 7fca76089640 1 librados: setting wanted keys 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.119+0000 7fca76089640 1 librados: calling monclient init 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.119+0000 7fca76089640 10 monclient: init 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 10 monclient(hunting): picked mon.noname-b con 0x7fca700a9470 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 10 monclient(hunting): picked mon.noname-a con 0x7fca700b10d0 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 10 monclient(hunting): picked mon.noname-c con 0x7fca700b4360 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca701db4b0 con 0x7fca700b10d0 2024-03-25T19:49:04.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700b04a0 con 0x7fca700a9470 2024-03-25T19:49:04.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700b0630 con 0x7fca700b4360 2024-03-25T19:49:04.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:04.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.120+0000 7fca76089640 10 monclient(hunting): authenticate will time out at 5350.432155s 2024-03-25T19:49:04.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.131+0000 7fca6effd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fca700b10d0 0x7fca7013efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60212/0 (socket says 172.21.15.71:60212) 2024-03-25T19:49:04.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.131+0000 7fca6effd640 1 -- 172.21.15.71:0/2124332867 learned_addr learned my addr 172.21.15.71:0/2124332867 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:04.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.132+0000 7fca57fff640 10 client.?.objecter ms_handle_connect 0x7fca700b10d0 2024-03-25T19:49:04.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.132+0000 7fca57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:04.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.133+0000 7fca57fff640 10 client.?.objecter ms_handle_connect 0x7fca700b4360 2024-03-25T19:49:04.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.133+0000 7fca57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:04.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.140+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684150768 0 0) 0x7fca700b0630 con 0x7fca700b4360 2024-03-25T19:49:04.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.140+0000 7fca57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:04.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.140+0000 7fca57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:04.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.140+0000 7fca57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:04.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.140+0000 7fca57fff640 10 monclient(hunting): my global_id is 14603 2024-03-25T19:49:04.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.140+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca5c004eb0 con 0x7fca700b4360 2024-03-25T19:49:04.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.141+0000 7fca57fff640 10 client.?.objecter ms_handle_connect 0x7fca700a9470 2024-03-25T19:49:04.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.141+0000 7fca57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:04.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.147+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398753387 0 0) 0x7fca701db4b0 con 0x7fca700b10d0 2024-03-25T19:49:04.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.147+0000 7fca57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:04.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.147+0000 7fca57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:04.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.147+0000 7fca57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:04.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.147+0000 7fca57fff640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:49:04.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.147+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca700b0630 con 0x7fca700b10d0 2024-03-25T19:49:04.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.156+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219900946 0 0) 0x7fca700b04a0 con 0x7fca700a9470 2024-03-25T19:49:04.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.156+0000 7fca57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:04.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.156+0000 7fca57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:04.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.156+0000 7fca57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:04.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.156+0000 7fca57fff640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:49:04.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.156+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca701db4b0 con 0x7fca700a9470 2024-03-25T19:49:04.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.166+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2157792044 0 0) 0x7fca701db4b0 con 0x7fca700a9470 2024-03-25T19:49:04.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.166+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca700b04a0 con 0x7fca700a9470 2024-03-25T19:49:04.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.166+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fca60001830 con 0x7fca700a9470 2024-03-25T19:49:04.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.166+0000 7fca57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:04.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.166+0000 7fca57fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:04.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.166+0000 7fca57fff640 10 monclient(hunting): dump: 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:04.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.166+0000 7fca57fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:04.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.170+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3253180961 0 0) 0x7fca700b0630 con 0x7fca700b10d0 2024-03-25T19:49:04.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.170+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca701db4b0 con 0x7fca700b10d0 2024-03-25T19:49:04.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.170+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fca640017f0 con 0x7fca700b10d0 2024-03-25T19:49:04.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.170+0000 7fca57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:04.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.170+0000 7fca57fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:04.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.170+0000 7fca57fff640 10 monclient(hunting): dump: 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:04.171 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:04.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 887895250 0 0) 0x7fca700b04a0 con 0x7fca700a9470 2024-03-25T19:49:04.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 >> v1:172.21.15.71:6790/0 conn(0x7fca700b4360 legacy=0x7fca70146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 >> v1:172.21.15.71:6789/0 conn(0x7fca700b10d0 legacy=0x7fca7013efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 1 monclient: found mon.b 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7013f6f0 con 0x7fca700a9470 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 5 monclient: authenticate success, global_id 4606 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 10 monclient: _renew_subs 2024-03-25T19:49:04.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7013f9c0 con 0x7fca700a9470 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 10 monclient: _renew_subs 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7014e940 con 0x7fca700a9470 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.172+0000 7fca76089640 1 librados: init done 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.173+0000 7fca76089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.173+0000 7fca76089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.173+0000 7fca76089640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:04.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.173+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca701db2f0 con 0x7fca700a9470 2024-03-25T19:49:04.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.175+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca60001d40 con 0x7fca700a9470 2024-03-25T19:49:04.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.175+0000 7fca57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:04.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.175+0000 7fca6d7fa640 4 set_mon_vals no callback set 2024-03-25T19:49:04.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.433+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fca60003610 con 0x7fca700a9470 2024-03-25T19:49:04.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.433+0000 7fca57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:04.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.433+0000 7fca57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:04.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.433+0000 7fca57fff640 10 monclient: dump: 2024-03-25T19:49:04.433 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:04.433 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:04.433 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.433 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:04.434 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:04.434 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:04.434 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:04.434 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:04.434 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:04.434 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:04.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.433+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fca60019660 con 0x7fca700a9470 2024-03-25T19:49:04.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.434+0000 7fca6effd640 10 monclient: get_auth_request con 0x7fca5c06df80 auth_method 0 2024-03-25T19:49:04.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.434+0000 7fca57fff640 10 client.4606.objecter ms_handle_connect 0x7fca5c06df80 2024-03-25T19:49:04.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.436+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fca60053440 con 0x7fca700a9470 2024-03-25T19:49:04.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.436+0000 7fca57fff640 10 client.4606.objecter ms_dispatch 0x7fca70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:04.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.436+0000 7fca57fff640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:04.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.436+0000 7fca57fff640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:04.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.436+0000 7fca57fff640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T19:49:04.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.439+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca6004ed80 con 0x7fca700a9470 2024-03-25T19:49:04.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.439+0000 7fca57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:04.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.439+0000 7fca57fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:04.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.688+0000 7fca76089640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:04.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.688+0000 7fca76089640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:04.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.688+0000 7fca76089640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:04.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.688+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca70148d20 con 0x7fca700a9470 2024-03-25T19:49:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.718+0000 7fca57fff640 1 -- 172.21.15.71:0/2124332867 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fca60050e80 con 0x7fca700a9470 2024-03-25T19:49:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.718+0000 7fca57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.718+0000 7fca57fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:04.719 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:49:04.719 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:04.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 10 librados: watch_flush enter 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 10 librados: watch_flush exit 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 10 client.4606.objecter successfully canceled tick 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 >> v1:172.21.15.71:6800/3911735624 conn(0x7fca5c06df80 legacy=0x7fca5c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 10 monclient: shutdown 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 >> v1:172.21.15.193:6789/0 conn(0x7fca700a9470 legacy=0x7fca701dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 shutdown_connections 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 >> 172.21.15.71:0/2124332867 conn(0x7fca700617a0 msgr2=0x7fca70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 shutdown_connections 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 1 -- 172.21.15.71:0/2124332867 wait complete. 2024-03-25T19:49:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:04.721+0000 7fca76089640 1 librados: shutdown 2024-03-25T19:49:04.733 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:11.539 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:11.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.655+0000 7f24a334d640 10 monclient: get_monmap_and_config 2024-03-25T19:49:11.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.655+0000 7f24a334d640 10 monclient: build_initial_monmap 2024-03-25T19:49:11.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.655+0000 7f24a334d640 10 monclient: monmap: 2024-03-25T19:49:11.655 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:11.655 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:11.655 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:49:11.656498+0000 2024-03-25T19:49:11.656 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:49:11.656498+0000 2024-03-25T19:49:11.656 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:11.656 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:11.656 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:11.656 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:49:11.656 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:49:11.656 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:11.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.656+0000 7f24a334d640 1 Processor -- start 2024-03-25T19:49:11.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.656+0000 7f24a334d640 1 -- start start 2024-03-25T19:49:11.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.656+0000 7f24a334d640 10 monclient: init 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 10 monclient(hunting): picked mon.noname-a con 0x7f249c0b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 10 monclient(hunting): picked mon.noname-b con 0x7f249c0b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 10 monclient(hunting): picked mon.noname-c con 0x7f249c14be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0af940 con 0x7f249c0b4d60 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0b0a70 con 0x7f249c14be90 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0b0d50 con 0x7f249c0b1730 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:11.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.657+0000 7f24a334d640 10 monclient(hunting): authenticate will time out at 5357.969078s 2024-03-25T19:49:11.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.670+0000 7f24a08e8640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f249c0b1730 0x7f249c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45152/0 (socket says 172.21.15.71:45152) 2024-03-25T19:49:11.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.670+0000 7f24a08e8640 1 -- 172.21.15.71:0/388700235 learned_addr learned my addr 172.21.15.71:0/388700235 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:11.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.686+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212683552 0 0) 0x7f249c0b0a70 con 0x7f249c14be90 2024-03-25T19:49:11.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.686+0000 7f2493fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:11.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.686+0000 7f2493fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:11.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.686+0000 7f2493fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:11.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.686+0000 7f2493fff640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:49:11.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.686+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2488003520 con 0x7f249c14be90 2024-03-25T19:49:11.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.688+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540560558 0 0) 0x7f249c0b0d50 con 0x7f249c0b1730 2024-03-25T19:49:11.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.688+0000 7f2493fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:11.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.688+0000 7f2493fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:11.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.688+0000 7f2493fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:11.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.688+0000 7f2493fff640 10 monclient(hunting): my global_id is 14606 2024-03-25T19:49:11.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.688+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c0b0a70 con 0x7f249c0b1730 2024-03-25T19:49:11.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.697+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156865474 0 0) 0x7f249c0af940 con 0x7f249c0b4d60 2024-03-25T19:49:11.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.697+0000 7f2493fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:11.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.697+0000 7f2493fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:11.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.697+0000 7f2493fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:11.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.697+0000 7f2493fff640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:49:11.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.697+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c0b0d50 con 0x7f249c0b4d60 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2430376623 0 0) 0x7f2488003520 con 0x7f249c14be90 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 >> v1:172.21.15.71:6790/0 conn(0x7f249c0b1730 legacy=0x7f249c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 >> v1:172.21.15.71:6789/0 conn(0x7f249c0b4d60 legacy=0x7f249c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 1 monclient: found mon.noname-c 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c0b1030 con 0x7f249c14be90 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:11.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2494001a10 con 0x7f249c14be90 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f2493fff640 10 monclient: dump: 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f24a334d640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:49:11.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.699+0000 7f24a334d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:11.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2494001bf0 con 0x7f249c14be90 2024-03-25T19:49:11.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f2493fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:11.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f2493fff640 1 -- 172.21.15.71:0/388700235 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2494002000 con 0x7f249c14be90 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f24a334d640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f24a334d640 4 set_mon_vals no callback set 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f2493fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f2493fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f2493fff640 10 monclient: dump: 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:11.705 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:11.706 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f24a334d640 10 monclient: shutdown 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f24a334d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f24a334d640 1 -- 172.21.15.71:0/388700235 >> v1:172.21.15.193:6789/0 conn(0x7f249c14be90 legacy=0x7f249c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f24a334d640 1 -- 172.21.15.71:0/388700235 shutdown_connections 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.705+0000 7f24a334d640 1 -- 172.21.15.71:0/388700235 >> 172.21.15.71:0/388700235 conn(0x7f249c0617a0 msgr2=0x7f249c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.706+0000 7f24a334d640 1 -- 172.21.15.71:0/388700235 shutdown_connections 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.706+0000 7f24a334d640 1 -- 172.21.15.71:0/388700235 wait complete. 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.706+0000 7f24a334d640 10 monclient: build_initial_monmap 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.706+0000 7f24a334d640 10 monclient: monmap: 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:11.708 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.707+0000 7f24a334d640 1 librados: starting msgr at 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.707+0000 7f24a334d640 1 librados: starting objecter 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.707+0000 7f24a334d640 1 Processor -- start 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.707+0000 7f24a334d640 1 -- start start 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.707+0000 7f24a334d640 1 librados: setting wanted keys 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.707+0000 7f24a334d640 1 librados: calling monclient init 2024-03-25T19:49:11.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.707+0000 7f24a334d640 10 monclient: init 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 10 monclient(hunting): picked mon.noname-a con 0x7f249c0b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 10 monclient(hunting): picked mon.noname-c con 0x7f249c0b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 10 monclient(hunting): picked mon.noname-b con 0x7f249c14be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c136380 con 0x7f249c0b1730 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c1376c0 con 0x7f249c14be90 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c1379a0 con 0x7f249c0b4d60 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:11.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.708+0000 7f24a334d640 10 monclient(hunting): authenticate will time out at 5358.020220s 2024-03-25T19:49:11.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.718+0000 7f24a18ea640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f249c14be90 0x7f249c140d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:46242/0 (socket says 172.21.15.71:46242) 2024-03-25T19:49:11.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.718+0000 7f24a18ea640 1 -- 172.21.15.71:0/4279508832 learned_addr learned my addr 172.21.15.71:0/4279508832 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:11.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.724+0000 7f2491ffb640 10 client.?.objecter ms_handle_connect 0x7f249c14be90 2024-03-25T19:49:11.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.724+0000 7f2491ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:11.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.735+0000 7f2491ffb640 10 client.?.objecter ms_handle_connect 0x7f249c0b4d60 2024-03-25T19:49:11.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.735+0000 7f2491ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:11.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.735+0000 7f2491ffb640 10 client.?.objecter ms_handle_connect 0x7f249c0b1730 2024-03-25T19:49:11.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.735+0000 7f2491ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:11.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.742+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382531273 0 0) 0x7f249c1376c0 con 0x7f249c14be90 2024-03-25T19:49:11.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.742+0000 7f2491ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:11.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.742+0000 7f2491ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:11.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.742+0000 7f2491ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:11.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.742+0000 7f2491ffb640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:49:11.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.742+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2480004eb0 con 0x7f249c14be90 2024-03-25T19:49:11.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.746+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306259892 0 0) 0x7f249c136380 con 0x7f249c0b1730 2024-03-25T19:49:11.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.746+0000 7f2491ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:11.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.746+0000 7f2491ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:11.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.746+0000 7f2491ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:11.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.746+0000 7f2491ffb640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:49:11.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.746+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c1376c0 con 0x7f249c0b1730 2024-03-25T19:49:11.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.755+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466355541 0 0) 0x7f249c1379a0 con 0x7f249c0b4d60 2024-03-25T19:49:11.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.755+0000 7f2491ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:11.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.755+0000 7f2491ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:11.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.755+0000 7f2491ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:11.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.755+0000 7f2491ffb640 10 monclient(hunting): my global_id is 14609 2024-03-25T19:49:11.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.755+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c136380 con 0x7f249c0b4d60 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.768+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1639730393 0 0) 0x7f249c1376c0 con 0x7f249c0b1730 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.768+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f249c1379a0 con 0x7f249c0b1730 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.768+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2498001870 con 0x7f249c0b1730 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.768+0000 7f2491ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.768+0000 7f2491ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.768+0000 7f2491ffb640 10 monclient(hunting): dump: 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:11.768 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.769 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.769 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:11.769 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:11.769 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:11.769 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:11.769 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:11.769 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:11.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.768+0000 7f2491ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:11.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.772+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278433973 0 0) 0x7f2480004eb0 con 0x7f249c14be90 2024-03-25T19:49:11.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.772+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f249c1376c0 con 0x7f249c14be90 2024-03-25T19:49:11.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.772+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f2494001f40 con 0x7f249c14be90 2024-03-25T19:49:11.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.772+0000 7f2491ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:11.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.772+0000 7f2491ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:11.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.772+0000 7f2491ffb640 10 monclient(hunting): dump: 2024-03-25T19:49:11.772 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:11.773 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.773+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1036153439 0 0) 0x7f249c136380 con 0x7f249c0b4d60 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.773+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2480004eb0 con 0x7f249c0b4d60 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.773+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f248c0018b0 con 0x7f249c0b4d60 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.773+0000 7f2491ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.773+0000 7f2491ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.773+0000 7f2491ffb640 10 monclient(hunting): dump: 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:11.774 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:11.775 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:11.775 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:11.775 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:11.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1289217788 0 0) 0x7f2480004eb0 con 0x7f249c0b4d60 2024-03-25T19:49:11.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 >> v1:172.21.15.193:6789/0 conn(0x7f249c14be90 legacy=0x7f249c140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:11.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 >> v1:172.21.15.71:6789/0 conn(0x7f249c0b1730 legacy=0x7f249c1427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:11.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 1 monclient: found mon.c 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c135fc0 con 0x7f249c0b4d60 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f2491ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f24a334d640 5 monclient: authenticate success, global_id 14609 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f24a334d640 10 monclient: _renew_subs 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f24a334d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.779+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c145620 con 0x7f249c0b4d60 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.780+0000 7f24a334d640 10 client.14609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.780+0000 7f24a334d640 10 monclient: _renew_subs 2024-03-25T19:49:11.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.780+0000 7f24a334d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:49:11.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.780+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c145970 con 0x7f249c0b4d60 2024-03-25T19:49:11.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.780+0000 7f24a334d640 1 librados: init done 2024-03-25T19:49:11.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.781+0000 7f24a334d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:11.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.781+0000 7f24a334d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:11.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.781+0000 7f24a334d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:49:11.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:11.781+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249c0627b0 con 0x7f249c0b4d60 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.092+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f248c0047e0 con 0x7f249c0b4d60 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.092+0000 7f2491ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.092+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f248c001d80 con 0x7f249c0b4d60 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.092+0000 7f2491ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.092+0000 7f2491ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.092+0000 7f2491ffb640 10 monclient: dump: 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:12.092 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:12.093 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:12.093 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:12.093 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:12.093 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:12.093 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:12.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.092+0000 7f24937fe640 4 set_mon_vals no callback set 2024-03-25T19:49:12.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.093+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f248c01a9a0 con 0x7f249c0b4d60 2024-03-25T19:49:12.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.094+0000 7f24a10e9640 10 monclient: get_auth_request con 0x7f248006e160 auth_method 0 2024-03-25T19:49:12.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.094+0000 7f2491ffb640 10 client.14609.objecter ms_handle_connect 0x7f248006e160 2024-03-25T19:49:12.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.099+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f248c0546a0 con 0x7f249c0b4d60 2024-03-25T19:49:12.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.099+0000 7f2491ffb640 10 client.14609.objecter ms_dispatch 0x7f249c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:12.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.099+0000 7f2491ffb640 3 client.14609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:12.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.099+0000 7f2491ffb640 3 client.14609.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:12.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.099+0000 7f2491ffb640 20 client.14609.objecter dump_active .. 0 homeless 2024-03-25T19:49:12.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.099+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f248c0501b0 con 0x7f249c0b4d60 2024-03-25T19:49:12.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.099+0000 7f2491ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:12.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.099+0000 7f2491ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:12.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.342+0000 7f24a334d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:12.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.342+0000 7f24a334d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:12.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.342+0000 7f24a334d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:49:12.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.342+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f249c136150 con 0x7f249c0b4d60 2024-03-25T19:49:12.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.369+0000 7f2491ffb640 1 -- 172.21.15.71:0/4279508832 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f249c136150 con 0x7f249c0b4d60 2024-03-25T19:49:12.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.369+0000 7f2491ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:12.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.369+0000 7f2491ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:12.369 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:49:12.369 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.371+0000 7f24a334d640 10 librados: watch_flush enter 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.371+0000 7f24a334d640 10 librados: watch_flush exit 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.371+0000 7f24a334d640 20 client.14609.objecter shutdown clearing up homeless session... 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.371+0000 7f24a334d640 10 client.14609.objecter successfully canceled tick 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.371+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 >> v1:172.21.15.71:6800/3911735624 conn(0x7f248006e160 legacy=0x7f2480070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.371+0000 7f24a334d640 10 monclient: shutdown 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.371+0000 7f24a334d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.371+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 >> v1:172.21.15.71:6790/0 conn(0x7f249c0b4d60 legacy=0x7f249c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.372+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 shutdown_connections 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.372+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 >> 172.21.15.71:0/4279508832 conn(0x7f249c0617a0 msgr2=0x7f249c13c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.372+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 shutdown_connections 2024-03-25T19:49:12.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.372+0000 7f24a334d640 1 -- 172.21.15.71:0/4279508832 wait complete. 2024-03-25T19:49:12.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:12.372+0000 7f24a334d640 1 librados: shutdown 2024-03-25T19:49:12.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:19.190 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:19.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.304+0000 7f8dea8ff640 10 monclient: get_monmap_and_config 2024-03-25T19:49:19.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.305+0000 7f8dea8ff640 10 monclient: build_initial_monmap 2024-03-25T19:49:19.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.305+0000 7f8dea8ff640 10 monclient: monmap: 2024-03-25T19:49:19.304 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:49:19.305917+0000 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:49:19.305917+0000 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.305+0000 7f8dea8ff640 1 Processor -- start 2024-03-25T19:49:19.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 1 -- start start 2024-03-25T19:49:19.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 10 monclient: init 2024-03-25T19:49:19.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:19.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:19.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 10 monclient(hunting): picked mon.noname-a con 0x7f8de40a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 10 monclient(hunting): picked mon.noname-c con 0x7f8de413f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 10 monclient(hunting): picked mon.noname-b con 0x7f8de41430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de4142520 con 0x7f8de40a98f0 2024-03-25T19:49:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40b28a0 con 0x7f8de413f710 2024-03-25T19:49:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40b2a30 con 0x7f8de41430c0 2024-03-25T19:49:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.306+0000 7f8dea8ff640 10 monclient(hunting): authenticate will time out at 5365.618796s 2024-03-25T19:49:19.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.320+0000 7f8de37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f8de413f710 0x7f8de41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:39066/0 (socket says 172.21.15.71:39066) 2024-03-25T19:49:19.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.320+0000 7f8de37fe640 1 -- 172.21.15.71:0/3163652700 learned_addr learned my addr 172.21.15.71:0/3163652700 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:19.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.330+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336169930 0 0) 0x7f8de4142520 con 0x7f8de40a98f0 2024-03-25T19:49:19.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.330+0000 7f8de2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:19.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.330+0000 7f8de2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:19.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.330+0000 7f8de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:19.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.330+0000 7f8de2ffd640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:49:19.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.330+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dcc003520 con 0x7f8de40a98f0 2024-03-25T19:49:19.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.337+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52320479 0 0) 0x7f8de40b28a0 con 0x7f8de413f710 2024-03-25T19:49:19.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.337+0000 7f8de2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:19.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.337+0000 7f8de2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:19.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.337+0000 7f8de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:19.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.337+0000 7f8de2ffd640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:49:19.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.337+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de4142520 con 0x7f8de413f710 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3504500364 0 0) 0x7f8dcc003520 con 0x7f8de40a98f0 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 >> v1:172.21.15.71:6790/0 conn(0x7f8de41430c0 legacy=0x7f8de414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 >> v1:172.21.15.193:6789/0 conn(0x7f8de413f710 legacy=0x7f8de41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 1 monclient: found mon.noname-a 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de40b44c0 con 0x7f8de40a98f0 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8dd4001690 con 0x7f8de40a98f0 2024-03-25T19:49:19.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8de2ffd640 10 monclient: dump: 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8dea8ff640 5 monclient: authenticate success, global_id 4617 2024-03-25T19:49:19.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.340+0000 7f8dea8ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.354+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dd40017f0 con 0x7f8de40a98f0 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.354+0000 7f8de2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.354+0000 7f8de2ffd640 1 -- 172.21.15.71:0/3163652700 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8dd4001c00 con 0x7f8de40a98f0 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.354+0000 7f8de2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.354+0000 7f8de2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.354+0000 7f8de2ffd640 10 monclient: dump: 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:19.354 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:19.355 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:19.355 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:19.355 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:19.355 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:19.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.354+0000 7f8dea8ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:19.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.354+0000 7f8dea8ff640 4 set_mon_vals no callback set 2024-03-25T19:49:19.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.355+0000 7f8dea8ff640 10 monclient: shutdown 2024-03-25T19:49:19.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.355+0000 7f8dea8ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:19.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.355+0000 7f8dea8ff640 1 -- 172.21.15.71:0/3163652700 >> v1:172.21.15.71:6789/0 conn(0x7f8de40a98f0 legacy=0x7f8de40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:19.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.355+0000 7f8dea8ff640 1 -- 172.21.15.71:0/3163652700 shutdown_connections 2024-03-25T19:49:19.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.355+0000 7f8dea8ff640 1 -- 172.21.15.71:0/3163652700 >> 172.21.15.71:0/3163652700 conn(0x7f8de40b7ea0 msgr2=0x7f8de40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.355+0000 7f8dea8ff640 1 -- 172.21.15.71:0/3163652700 shutdown_connections 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.355+0000 7f8dea8ff640 1 -- 172.21.15.71:0/3163652700 wait complete. 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.356+0000 7f8dea8ff640 10 monclient: build_initial_monmap 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.356+0000 7f8dea8ff640 10 monclient: monmap: 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.356+0000 7f8dea8ff640 1 librados: starting msgr at 2024-03-25T19:49:19.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.356+0000 7f8dea8ff640 1 librados: starting objecter 2024-03-25T19:49:19.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.356+0000 7f8dea8ff640 1 Processor -- start 2024-03-25T19:49:19.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 1 -- start start 2024-03-25T19:49:19.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 1 librados: setting wanted keys 2024-03-25T19:49:19.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 1 librados: calling monclient init 2024-03-25T19:49:19.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 10 monclient: init 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 10 monclient(hunting): picked mon.noname-a con 0x7f8de40a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 10 monclient(hunting): picked mon.noname-c con 0x7f8de413f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 10 monclient(hunting): picked mon.noname-b con 0x7f8de41430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de4132a30 con 0x7f8de40a98f0 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de4138c80 con 0x7f8de41430c0 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de4138f60 con 0x7f8de413f710 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:19.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.357+0000 7f8dea8ff640 10 monclient(hunting): authenticate will time out at 5365.669763s 2024-03-25T19:49:19.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.362+0000 7f8de3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8de40a98f0 0x7f8de4137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54230/0 (socket says 172.21.15.71:54230) 2024-03-25T19:49:19.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.362+0000 7f8de3fff640 1 -- 172.21.15.71:0/818576766 learned_addr learned my addr 172.21.15.71:0/818576766 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:19.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.365+0000 7f8de0ff9640 10 client.?.objecter ms_handle_connect 0x7f8de41430c0 2024-03-25T19:49:19.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.365+0000 7f8de0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:19.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.372+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261903324 0 0) 0x7f8de4138c80 con 0x7f8de41430c0 2024-03-25T19:49:19.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.372+0000 7f8de0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:19.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.372+0000 7f8de0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:19.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.372+0000 7f8de0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:19.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.372+0000 7f8de0ff9640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:49:19.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.372+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd0004eb0 con 0x7f8de41430c0 2024-03-25T19:49:19.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.372+0000 7f8de0ff9640 10 client.?.objecter ms_handle_connect 0x7f8de413f710 2024-03-25T19:49:19.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.372+0000 7f8de0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:19.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.376+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721856837 0 0) 0x7f8de4138f60 con 0x7f8de413f710 2024-03-25T19:49:19.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.376+0000 7f8de0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:19.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.376+0000 7f8de0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:19.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.376+0000 7f8de0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:19.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.376+0000 7f8de0ff9640 10 monclient(hunting): my global_id is 14615 2024-03-25T19:49:19.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.376+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de4138c80 con 0x7f8de413f710 2024-03-25T19:49:19.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.380+0000 7f8de0ff9640 10 client.?.objecter ms_handle_connect 0x7f8de40a98f0 2024-03-25T19:49:19.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.380+0000 7f8de0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:19.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.381+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4045611038 0 0) 0x7f8dd0004eb0 con 0x7f8de41430c0 2024-03-25T19:49:19.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.381+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8de4138f60 con 0x7f8de41430c0 2024-03-25T19:49:19.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.381+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8ddc001b20 con 0x7f8de41430c0 2024-03-25T19:49:19.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.381+0000 7f8de0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:19.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.381+0000 7f8de0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:19.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.381+0000 7f8de0ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:19.381 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:19.381 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:19.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.381+0000 7f8de0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:19.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.390+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3658963411 0 0) 0x7f8de4138c80 con 0x7f8de413f710 2024-03-25T19:49:19.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.390+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dd0004eb0 con 0x7f8de413f710 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.390+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8dd8001930 con 0x7f8de413f710 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.390+0000 7f8de0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.390+0000 7f8de0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.390+0000 7f8de0ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:19.390 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2049645594 0 0) 0x7f8de4138f60 con 0x7f8de41430c0 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 >> v1:172.21.15.71:6790/0 conn(0x7f8de413f710 legacy=0x7f8de4131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 >> v1:172.21.15.71:6789/0 conn(0x7f8de40a98f0 legacy=0x7f8de4137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 1 monclient: found mon.b 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4139240 con 0x7f8de41430c0 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8de0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 10 monclient: _renew_subs 2024-03-25T19:49:19.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de4136e50 con 0x7f8de41430c0 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 10 monclient: _renew_subs 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de41389f0 con 0x7f8de41430c0 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.391+0000 7f8dea8ff640 1 librados: init done 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.393+0000 7f8dea8ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.393+0000 7f8dea8ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:19.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.393+0000 7f8dea8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:19.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.393+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de40627b0 con 0x7f8de41430c0 2024-03-25T19:49:19.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.420+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ddc002030 con 0x7f8de41430c0 2024-03-25T19:49:19.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.420+0000 7f8de0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:19.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.420+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8ddc003b10 con 0x7f8de41430c0 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.420+0000 7f8de0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.420+0000 7f8de0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.420+0000 7f8de0ff9640 10 monclient: dump: 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:19.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.420+0000 7f8de27fc640 4 set_mon_vals no callback set 2024-03-25T19:49:19.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.421+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f8ddc019ae0 con 0x7f8de41430c0 2024-03-25T19:49:19.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.421+0000 7f8de3fff640 10 monclient: get_auth_request con 0x7f8dd006de40 auth_method 0 2024-03-25T19:49:19.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.421+0000 7f8de0ff9640 10 client.4618.objecter ms_handle_connect 0x7f8dd006de40 2024-03-25T19:49:19.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.423+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f8ddc053800 con 0x7f8de41430c0 2024-03-25T19:49:19.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.423+0000 7f8de0ff9640 10 client.4618.objecter ms_dispatch 0x7f8de4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:19.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.423+0000 7f8de0ff9640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:19.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.423+0000 7f8de0ff9640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:19.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.423+0000 7f8de0ff9640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T19:49:19.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.691+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ddc04f140 con 0x7f8de41430c0 2024-03-25T19:49:19.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.691+0000 7f8de0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:19.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.691+0000 7f8de0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:19.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.945+0000 7f8dea8ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:19.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.945+0000 7f8dea8ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:19.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.945+0000 7f8dea8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:19.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.945+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8de41378f0 con 0x7f8de41430c0 2024-03-25T19:49:19.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.974+0000 7f8de0ff9640 1 -- 172.21.15.71:0/818576766 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f8de41378f0 con 0x7f8de41430c0 2024-03-25T19:49:19.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.974+0000 7f8de0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:19.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.974+0000 7f8de0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:19.974 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:49:19.975 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.976+0000 7f8dea8ff640 10 librados: watch_flush enter 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 10 librados: watch_flush exit 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 10 client.4618.objecter successfully canceled tick 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 >> v1:172.21.15.71:6800/3911735624 conn(0x7f8dd006de40 legacy=0x7f8dd0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 10 monclient: shutdown 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 >> v1:172.21.15.193:6789/0 conn(0x7f8de41430c0 legacy=0x7f8de4132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 shutdown_connections 2024-03-25T19:49:19.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 >> 172.21.15.71:0/818576766 conn(0x7f8de40b7ea0 msgr2=0x7f8de4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:19.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 shutdown_connections 2024-03-25T19:49:19.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 1 -- 172.21.15.71:0/818576766 wait complete. 2024-03-25T19:49:19.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:19.977+0000 7f8dea8ff640 1 librados: shutdown 2024-03-25T19:49:19.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:22.894 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:05.650 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.904 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:05.660 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.937 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:05.693 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.957 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:05.676 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.963 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:05.719 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.964 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:05.684 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.974 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:05.731 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.995 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:05.751 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.000 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:05.719 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:26.794 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:26.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.909+0000 7f0045ef4640 10 monclient: get_monmap_and_config 2024-03-25T19:49:26.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.909+0000 7f0045ef4640 10 monclient: build_initial_monmap 2024-03-25T19:49:26.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.909+0000 7f0045ef4640 10 monclient: monmap: 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:49:26.910370+0000 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:49:26.910370+0000 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:49:26.910 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:26.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.910+0000 7f0045ef4640 1 Processor -- start 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.910+0000 7f0045ef4640 1 -- start start 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.910+0000 7f0045ef4640 10 monclient: init 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f00400a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f004013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 10 monclient(hunting): picked mon.noname-b con 0x7f00401430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0040142520 con 0x7f00400a98f0 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00400b28a0 con 0x7f004013f710 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00400b2a30 con 0x7f00401430c0 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:26.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.911+0000 7f0045ef4640 10 monclient(hunting): authenticate will time out at 5373.223116s 2024-03-25T19:49:26.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.918+0000 7f003f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f00400a98f0 0x7f00400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42486/0 (socket says 172.21.15.71:42486) 2024-03-25T19:49:26.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.918+0000 7f003f7fe640 1 -- 172.21.15.71:0/639694099 learned_addr learned my addr 172.21.15.71:0/639694099 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:26.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.936+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160844852 0 0) 0x7f0040142520 con 0x7f00400a98f0 2024-03-25T19:49:26.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.936+0000 7f003e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:26.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.936+0000 7f003e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:26.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.936+0000 7f003e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:26.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.936+0000 7f003e7fc640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:49:26.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.936+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0028003520 con 0x7f00400a98f0 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.950+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3648379620 0 0) 0x7f0028003520 con 0x7f00400a98f0 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 >> v1:172.21.15.71:6790/0 conn(0x7f00401430c0 legacy=0x7f004014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 >> v1:172.21.15.193:6789/0 conn(0x7f004013f710 legacy=0x7f00401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00400b44c0 con 0x7f00400a98f0 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:26.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f0030001690 con 0x7f00400a98f0 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f003e7fc640 10 monclient: dump: 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:26.952 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:26.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f0045ef4640 5 monclient: authenticate success, global_id 4623 2024-03-25T19:49:26.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.951+0000 7f0045ef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:26.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.971+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00300017f0 con 0x7f00400a98f0 2024-03-25T19:49:26.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.971+0000 7f003e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:26.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.971+0000 7f003e7fc640 1 -- 172.21.15.71:0/639694099 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f0030001c00 con 0x7f00400a98f0 2024-03-25T19:49:26.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.971+0000 7f0045ef4640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:26.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.971+0000 7f0045ef4640 4 set_mon_vals no callback set 2024-03-25T19:49:26.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.971+0000 7f003e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.971+0000 7f003e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.971+0000 7f003e7fc640 10 monclient: dump: 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:26.972 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:26.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.972+0000 7f0045ef4640 10 monclient: shutdown 2024-03-25T19:49:26.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.972+0000 7f0045ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.972+0000 7f0045ef4640 1 -- 172.21.15.71:0/639694099 >> v1:172.21.15.71:6789/0 conn(0x7f00400a98f0 legacy=0x7f00400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.972+0000 7f0045ef4640 1 -- 172.21.15.71:0/639694099 shutdown_connections 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.972+0000 7f0045ef4640 1 -- 172.21.15.71:0/639694099 >> 172.21.15.71:0/639694099 conn(0x7f00400b7ea0 msgr2=0x7f00400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.972+0000 7f0045ef4640 1 -- 172.21.15.71:0/639694099 shutdown_connections 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.972+0000 7f0045ef4640 1 -- 172.21.15.71:0/639694099 wait complete. 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 10 monclient: build_initial_monmap 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 10 monclient: monmap: 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:49:26.975 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 1 librados: starting msgr at 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 1 librados: starting objecter 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 1 Processor -- start 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 1 -- start start 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 1 librados: setting wanted keys 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 1 librados: calling monclient init 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.973+0000 7f0045ef4640 10 monclient: init 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f00400a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 10 monclient(hunting): picked mon.noname-b con 0x7f004013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:26.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f00401430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:26.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00400abd30 con 0x7f00401430c0 2024-03-25T19:49:26.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00401d77a0 con 0x7f004013f710 2024-03-25T19:49:26.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00401d51f0 con 0x7f00400a98f0 2024-03-25T19:49:26.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:26.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.974+0000 7f0045ef4640 10 monclient(hunting): authenticate will time out at 5373.286329s 2024-03-25T19:49:26.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.979+0000 7f003f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f00400a98f0 0x7f00401d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55604/0 (socket says 172.21.15.71:55604) 2024-03-25T19:49:26.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.979+0000 7f003f7fe640 1 -- 172.21.15.71:0/531554936 learned_addr learned my addr 172.21.15.71:0/531554936 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:26.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.980+0000 7f0027fff640 10 client.?.objecter ms_handle_connect 0x7f00400a98f0 2024-03-25T19:49:26.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.980+0000 7f0027fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:26.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.987+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56366877 0 0) 0x7f00401d51f0 con 0x7f00400a98f0 2024-03-25T19:49:26.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.987+0000 7f0027fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:26.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.987+0000 7f0027fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:26.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.987+0000 7f0027fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:26.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.987+0000 7f0027fff640 10 monclient(hunting): my global_id is 14621 2024-03-25T19:49:26.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.987+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f002c004eb0 con 0x7f00400a98f0 2024-03-25T19:49:26.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.988+0000 7f0027fff640 10 client.?.objecter ms_handle_connect 0x7f004013f710 2024-03-25T19:49:26.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.988+0000 7f0027fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:26.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.993+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638989740 0 0) 0x7f00401d77a0 con 0x7f004013f710 2024-03-25T19:49:26.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.993+0000 7f0027fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:26.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.993+0000 7f0027fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:26.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.993+0000 7f0027fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:26.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.993+0000 7f0027fff640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:49:26.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:26.993+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00401d51f0 con 0x7f004013f710 2024-03-25T19:49:26.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.000+0000 7f0027fff640 10 client.?.objecter ms_handle_connect 0x7f00401430c0 2024-03-25T19:49:26.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.000+0000 7f0027fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.003+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2365262897 0 0) 0x7f00401d51f0 con 0x7f004013f710 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.003+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00401d77a0 con 0x7f004013f710 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.003+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f0034001930 con 0x7f004013f710 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.003+0000 7f0027fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.003+0000 7f0027fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.003+0000 7f0027fff640 10 monclient(hunting): dump: 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:27.003 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:27.004 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:27.004 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:27.004 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:27.004 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:27.004 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:27.004 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:27.004 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:27.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.003+0000 7f0027fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:27.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.008+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1982018605 0 0) 0x7f002c004eb0 con 0x7f00400a98f0 2024-03-25T19:49:27.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.008+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00401d51f0 con 0x7f00400a98f0 2024-03-25T19:49:27.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.008+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f0030001c00 con 0x7f00400a98f0 2024-03-25T19:49:27.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.008+0000 7f0027fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:27.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.008+0000 7f0027fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:27.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.008+0000 7f0027fff640 10 monclient(hunting): dump: 2024-03-25T19:49:27.008 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:27.008 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:27.009 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:27.009 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:27.009 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:27.009 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:27.009 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:27.009 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:27.009 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:27.009 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4159985830 0 0) 0x7f00401d77a0 con 0x7f004013f710 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 >> v1:172.21.15.71:6790/0 conn(0x7f00400a98f0 legacy=0x7f00401d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 >> v1:172.21.15.71:6789/0 conn(0x7f00401430c0 legacy=0x7f00401e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 1 monclient: found mon.b 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00401d83d0 con 0x7f004013f710 2024-03-25T19:49:27.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0027fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 5 monclient: authenticate success, global_id 4624 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 10 monclient: _renew_subs 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0040138a50 con 0x7f004013f710 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 10 monclient: _renew_subs 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00401e35c0 con 0x7f004013f710 2024-03-25T19:49:27.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.011+0000 7f0045ef4640 1 librados: init done 2024-03-25T19:49:27.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.012+0000 7f0045ef4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:27.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.012+0000 7f0045ef4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:27.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.012+0000 7f0045ef4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:27.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.012+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00400627b0 con 0x7f004013f710 2024-03-25T19:49:27.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.014+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0034003610 con 0x7f004013f710 2024-03-25T19:49:27.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.014+0000 7f0027fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:27.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.014+0000 7f003dffb640 4 set_mon_vals no callback set 2024-03-25T19:49:27.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.277+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f0034003960 con 0x7f004013f710 2024-03-25T19:49:27.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.277+0000 7f0027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:27.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.277+0000 7f0027fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.277+0000 7f0027fff640 10 monclient: dump: 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:27.278 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:27.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.278+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f0034003ac0 con 0x7f004013f710 2024-03-25T19:49:27.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.279+0000 7f003f7fe640 10 monclient: get_auth_request con 0x7f002c06de40 auth_method 0 2024-03-25T19:49:27.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.279+0000 7f0027fff640 10 client.4624.objecter ms_handle_connect 0x7f002c06de40 2024-03-25T19:49:27.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.282+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f0034053640 con 0x7f004013f710 2024-03-25T19:49:27.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.282+0000 7f0027fff640 10 client.4624.objecter ms_dispatch 0x7f0040000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:27.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.282+0000 7f0027fff640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:27.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.282+0000 7f0027fff640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:27.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.282+0000 7f0027fff640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T19:49:27.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.285+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f003404f150 con 0x7f004013f710 2024-03-25T19:49:27.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.285+0000 7f0027fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:27.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.285+0000 7f0027fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:27.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.529+0000 7f0045ef4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:27.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.529+0000 7f0045ef4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:27.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.529+0000 7f0045ef4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:27.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.529+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f004013e250 con 0x7f004013f710 2024-03-25T19:49:27.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.560+0000 7f0027fff640 1 -- 172.21.15.71:0/531554936 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f0034051080 con 0x7f004013f710 2024-03-25T19:49:27.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.560+0000 7f0027fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:27.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.560+0000 7f0027fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:27.560 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:49:27.560 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:27.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 10 librados: watch_flush enter 2024-03-25T19:49:27.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 10 librados: watch_flush exit 2024-03-25T19:49:27.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T19:49:27.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 10 client.4624.objecter successfully canceled tick 2024-03-25T19:49:27.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 >> v1:172.21.15.71:6800/3911735624 conn(0x7f002c06de40 legacy=0x7f002c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:27.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 10 monclient: shutdown 2024-03-25T19:49:27.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:27.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 >> v1:172.21.15.193:6789/0 conn(0x7f004013f710 legacy=0x7f0040138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 shutdown_connections 2024-03-25T19:49:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 >> 172.21.15.71:0/531554936 conn(0x7f00400b7ea0 msgr2=0x7f0040074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 shutdown_connections 2024-03-25T19:49:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.562+0000 7f0045ef4640 1 -- 172.21.15.71:0/531554936 wait complete. 2024-03-25T19:49:27.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:27.563+0000 7f0045ef4640 1 librados: shutdown 2024-03-25T19:49:27.575 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:34.382 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:34.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.497+0000 7f680ad0d640 10 monclient: get_monmap_and_config 2024-03-25T19:49:34.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.497+0000 7f680ad0d640 10 monclient: build_initial_monmap 2024-03-25T19:49:34.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.497+0000 7f680ad0d640 10 monclient: monmap: 2024-03-25T19:49:34.498 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:49:34.499631+0000 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:49:34.499631+0000 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:49:34.499 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.498+0000 7f680ad0d640 1 Processor -- start 2024-03-25T19:49:34.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.498+0000 7f680ad0d640 1 -- start start 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.498+0000 7f680ad0d640 10 monclient: init 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 10 monclient(hunting): picked mon.noname-a con 0x7f68040b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 10 monclient(hunting): picked mon.noname-b con 0x7f68040b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 10 monclient(hunting): picked mon.noname-c con 0x7f680414be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68040af940 con 0x7f68040b4d60 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68040b0a70 con 0x7f680414be90 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68040b0d50 con 0x7f68040b1730 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.499+0000 7f680ad0d640 10 monclient(hunting): authenticate will time out at 5380.812221s 2024-03-25T19:49:34.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.509+0000 7f6803fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f68040b1730 0x7f68040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:60450/0 (socket says 172.21.15.71:60450) 2024-03-25T19:49:34.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.509+0000 7f6803fff640 1 -- 172.21.15.71:0/104827008 learned_addr learned my addr 172.21.15.71:0/104827008 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:34.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.528+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402784675 0 0) 0x7f68040af940 con 0x7f68040b4d60 2024-03-25T19:49:34.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.528+0000 7f68037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:34.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.528+0000 7f68037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:34.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.528+0000 7f68037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:34.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.528+0000 7f68037fe640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:49:34.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.528+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f4003520 con 0x7f68040b4d60 2024-03-25T19:49:34.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.534+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216272273 0 0) 0x7f68040b0a70 con 0x7f680414be90 2024-03-25T19:49:34.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.534+0000 7f68037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:34.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.534+0000 7f68037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:34.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.534+0000 7f68037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:34.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.534+0000 7f68037fe640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:49:34.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.534+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68040af940 con 0x7f680414be90 2024-03-25T19:49:34.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.535+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151327104 0 0) 0x7f68040b0d50 con 0x7f68040b1730 2024-03-25T19:49:34.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.535+0000 7f68037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:34.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.535+0000 7f68037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:34.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.535+0000 7f68037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:34.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.535+0000 7f68037fe640 10 monclient(hunting): my global_id is 14624 2024-03-25T19:49:34.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.535+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68040b0a70 con 0x7f68040b1730 2024-03-25T19:49:34.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 627255364 0 0) 0x7f68040af940 con 0x7f680414be90 2024-03-25T19:49:34.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 >> v1:172.21.15.71:6790/0 conn(0x7f68040b1730 legacy=0x7f68040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:34.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 >> v1:172.21.15.71:6789/0 conn(0x7f68040b4d60 legacy=0x7f680413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 1 monclient: found mon.noname-c 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68040b1030 con 0x7f680414be90 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f67fc0019f0 con 0x7f680414be90 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:34.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f68037fe640 10 monclient: dump: 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f680ad0d640 5 monclient: authenticate success, global_id 4627 2024-03-25T19:49:34.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.546+0000 7f680ad0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.551+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67fc001b50 con 0x7f680414be90 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.551+0000 7f68037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.551+0000 7f68037fe640 1 -- 172.21.15.71:0/104827008 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f67fc001f60 con 0x7f680414be90 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.551+0000 7f68037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.551+0000 7f68037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.551+0000 7f68037fe640 10 monclient: dump: 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.552 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.553 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:34.553 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:34.553 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:34.553 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:34.553 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:34.553 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:34.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.551+0000 7f680ad0d640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:34.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.551+0000 7f680ad0d640 4 set_mon_vals no callback set 2024-03-25T19:49:34.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 10 monclient: shutdown 2024-03-25T19:49:34.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:34.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 1 -- 172.21.15.71:0/104827008 >> v1:172.21.15.193:6789/0 conn(0x7f680414be90 legacy=0x7f680414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:34.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 1 -- 172.21.15.71:0/104827008 shutdown_connections 2024-03-25T19:49:34.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 1 -- 172.21.15.71:0/104827008 >> 172.21.15.71:0/104827008 conn(0x7f68040617a0 msgr2=0x7f6804061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:34.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 1 -- 172.21.15.71:0/104827008 shutdown_connections 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 1 -- 172.21.15.71:0/104827008 wait complete. 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 10 monclient: build_initial_monmap 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.552+0000 7f680ad0d640 10 monclient: monmap: 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:49:34.556 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.553+0000 7f680ad0d640 1 librados: starting msgr at 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.553+0000 7f680ad0d640 1 librados: starting objecter 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.553+0000 7f680ad0d640 1 Processor -- start 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.553+0000 7f680ad0d640 1 -- start start 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.553+0000 7f680ad0d640 1 librados: setting wanted keys 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.553+0000 7f680ad0d640 1 librados: calling monclient init 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.553+0000 7f680ad0d640 10 monclient: init 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 10 monclient(hunting): picked mon.noname-c con 0x7f68040b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 10 monclient(hunting): picked mon.noname-a con 0x7f68040b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 10 monclient(hunting): picked mon.noname-b con 0x7f680414be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:34.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6804141ff0 con 0x7f68040b4d60 2024-03-25T19:49:34.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68041422d0 con 0x7f680414be90 2024-03-25T19:49:34.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6804145630 con 0x7f68040b1730 2024-03-25T19:49:34.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:34.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.554+0000 7f680ad0d640 10 monclient(hunting): authenticate will time out at 5380.867197s 2024-03-25T19:49:34.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.560+0000 7f6803fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f68040b4d60 0x7f6804130fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47392/0 (socket says 172.21.15.71:47392) 2024-03-25T19:49:34.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.560+0000 7f6803fff640 1 -- 172.21.15.71:0/3004278106 learned_addr learned my addr 172.21.15.71:0/3004278106 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:34.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.562+0000 7f68017fa640 10 client.?.objecter ms_handle_connect 0x7f68040b4d60 2024-03-25T19:49:34.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.562+0000 7f68017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:34.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.567+0000 7f68017fa640 10 client.?.objecter ms_handle_connect 0x7f68040b1730 2024-03-25T19:49:34.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.567+0000 7f68017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:34.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.568+0000 7f68017fa640 10 client.?.objecter ms_handle_connect 0x7f680414be90 2024-03-25T19:49:34.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.568+0000 7f68017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:34.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.572+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634199701 0 0) 0x7f6804141ff0 con 0x7f68040b4d60 2024-03-25T19:49:34.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.572+0000 7f68017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:34.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.572+0000 7f68017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:34.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.572+0000 7f68017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:34.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.572+0000 7f68017fa640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:49:34.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.572+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e8004eb0 con 0x7f68040b4d60 2024-03-25T19:49:34.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.581+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216777263 0 0) 0x7f6804145630 con 0x7f68040b1730 2024-03-25T19:49:34.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.581+0000 7f68017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:34.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.581+0000 7f68017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:34.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.581+0000 7f68017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:34.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.581+0000 7f68017fa640 10 monclient(hunting): my global_id is 14627 2024-03-25T19:49:34.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.581+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6804141ff0 con 0x7f68040b1730 2024-03-25T19:49:34.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.594+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1425317081 0 0) 0x7f67e8004eb0 con 0x7f68040b4d60 2024-03-25T19:49:34.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.594+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6804145630 con 0x7f68040b4d60 2024-03-25T19:49:34.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.594+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f67f0001910 con 0x7f68040b4d60 2024-03-25T19:49:34.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.594+0000 7f68017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:34.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.594+0000 7f68017fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:34.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.594+0000 7f68017fa640 10 monclient(hunting): dump: 2024-03-25T19:49:34.595 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:34.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.594+0000 7f68017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:34.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.597+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193515002 0 0) 0x7f68041422d0 con 0x7f680414be90 2024-03-25T19:49:34.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.597+0000 7f68017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:34.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.597+0000 7f68017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:34.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.597+0000 7f68017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:34.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.597+0000 7f68017fa640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:49:34.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.597+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e8004eb0 con 0x7f680414be90 2024-03-25T19:49:34.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.602+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1595402871 0 0) 0x7f6804141ff0 con 0x7f68040b1730 2024-03-25T19:49:34.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.602+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68041422d0 con 0x7f68040b1730 2024-03-25T19:49:34.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.603+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f67f8001950 con 0x7f68040b1730 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.603+0000 7f68017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.603+0000 7f68017fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.603+0000 7f68017fa640 10 monclient(hunting): dump: 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:34.604 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:34.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.609+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2884619567 0 0) 0x7f67e8004eb0 con 0x7f680414be90 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.610+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6804141ff0 con 0x7f680414be90 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.610+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f67fc001e60 con 0x7f680414be90 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.610+0000 7f68017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.610+0000 7f68017fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.610+0000 7f68017fa640 10 monclient(hunting): dump: 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:34.611 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:34.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1112072096 0 0) 0x7f6804145630 con 0x7f68040b4d60 2024-03-25T19:49:34.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 >> v1:172.21.15.71:6790/0 conn(0x7f68040b1730 legacy=0x7f6804140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:34.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 >> v1:172.21.15.193:6789/0 conn(0x7f680414be90 legacy=0x7f68041e3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:34.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 10 monclient: _finish_hunting 0 2024-03-25T19:49:34.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 1 monclient: found mon.a 2024-03-25T19:49:34.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:34.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68041452c0 con 0x7f68040b4d60 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 10 monclient: _finish_auth 0 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 10 monclient: _check_auth_tickets 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f68017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 5 monclient: authenticate success, global_id 4632 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 10 monclient: _renew_subs 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6804140fb0 con 0x7f68040b4d60 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 10 client.4632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 10 monclient: _renew_subs 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6804140d70 con 0x7f68040b4d60 2024-03-25T19:49:34.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.621+0000 7f680ad0d640 1 librados: init done 2024-03-25T19:49:34.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.622+0000 7f680ad0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:34.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.622+0000 7f680ad0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:34.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.622+0000 7f680ad0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:49:34.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.622+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68041457c0 con 0x7f68040b4d60 2024-03-25T19:49:34.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.863+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f00047e0 con 0x7f68040b4d60 2024-03-25T19:49:34.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.863+0000 7f68017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:34.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.863+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f67f0004bf0 con 0x7f68040b4d60 2024-03-25T19:49:34.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.863+0000 7f68017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:34.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.863+0000 7f68017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:34.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.863+0000 7f68017fa640 10 monclient: dump: 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:34.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.863+0000 7f6802ffd640 4 set_mon_vals no callback set 2024-03-25T19:49:34.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.864+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f67f0004e10 con 0x7f68040b4d60 2024-03-25T19:49:34.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.864+0000 7f6808aa9640 10 monclient: get_auth_request con 0x7f67e806e0e0 auth_method 0 2024-03-25T19:49:34.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.864+0000 7f68017fa640 10 client.4632.objecter ms_handle_connect 0x7f67e806e0e0 2024-03-25T19:49:34.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.869+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f67f00547c0 con 0x7f68040b4d60 2024-03-25T19:49:34.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.869+0000 7f68017fa640 10 client.4632.objecter ms_dispatch 0x7f6804000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:34.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.869+0000 7f68017fa640 3 client.4632.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:34.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.869+0000 7f68017fa640 3 client.4632.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:34.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.869+0000 7f68017fa640 20 client.4632.objecter dump_active .. 0 homeless 2024-03-25T19:49:34.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.869+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67f00502d0 con 0x7f68040b4d60 2024-03-25T19:49:34.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.869+0000 7f68017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:34.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:34.869+0000 7f68017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:35.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.091+0000 7f680ad0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:35.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.092+0000 7f680ad0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:35.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.092+0000 7f680ad0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:49:35.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.092+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68040627b0 con 0x7f68040b4d60 2024-03-25T19:49:35.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.120+0000 7f68017fa640 1 -- 172.21.15.71:0/3004278106 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f67f0052200 con 0x7f68040b4d60 2024-03-25T19:49:35.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.120+0000 7f68017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:35.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.120+0000 7f68017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:35.121 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:49:35.121 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:35.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 10 librados: watch_flush enter 2024-03-25T19:49:35.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 10 librados: watch_flush exit 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 20 client.4632.objecter shutdown clearing up homeless session... 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 10 client.4632.objecter successfully canceled tick 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 >> v1:172.21.15.71:6800/3911735624 conn(0x7f67e806e0e0 legacy=0x7f67e80705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 10 monclient: shutdown 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 >> v1:172.21.15.71:6789/0 conn(0x7f68040b4d60 legacy=0x7f6804130fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 shutdown_connections 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 >> 172.21.15.71:0/3004278106 conn(0x7f68040617a0 msgr2=0x7f680413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 shutdown_connections 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.122+0000 7f680ad0d640 1 -- 172.21.15.71:0/3004278106 wait complete. 2024-03-25T19:49:35.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:35.123+0000 7f680ad0d640 1 librados: shutdown 2024-03-25T19:49:35.134 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:41.941 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:42.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.054+0000 7f5aaea06640 10 monclient: get_monmap_and_config 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.054+0000 7f5aaea06640 10 monclient: build_initial_monmap 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.054+0000 7f5aaea06640 10 monclient: monmap: 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:49:42.056722+0000 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:49:42.056722+0000 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:49:42.056 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:42.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.056+0000 7f5aaea06640 1 Processor -- start 2024-03-25T19:49:42.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.056+0000 7f5aaea06640 1 -- start start 2024-03-25T19:49:42.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.056+0000 7f5aaea06640 10 monclient: init 2024-03-25T19:49:42.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:42.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:42.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 10 monclient(hunting): picked mon.noname-c con 0x7f5aa80a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 10 monclient(hunting): picked mon.noname-a con 0x7f5aa813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 10 monclient(hunting): picked mon.noname-b con 0x7f5aa81430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa8142520 con 0x7f5aa813f710 2024-03-25T19:49:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa80b28a0 con 0x7f5aa80a98f0 2024-03-25T19:49:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa80b2a30 con 0x7f5aa81430c0 2024-03-25T19:49:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.057+0000 7f5aaea06640 10 monclient(hunting): authenticate will time out at 5388.370096s 2024-03-25T19:49:42.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.064+0000 7f5aa77fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5aa813f710 0x7f5aa81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:45770/0 (socket says 172.21.15.71:45770) 2024-03-25T19:49:42.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.064+0000 7f5aa77fe640 1 -- 172.21.15.71:0/3003024874 learned_addr learned my addr 172.21.15.71:0/3003024874 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:42.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.071+0000 7f5aacfa3640 1 -- 172.21.15.71:0/3003024874 >> v1:172.21.15.71:6790/0 conn(0x7f5aa81430c0 legacy=0x7f5aa814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T19:49:42.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.071+0000 7f5aacfa3640 1 -- 172.21.15.71:0/3003024874 >> v1:172.21.15.71:6790/0 conn(0x7f5aa81430c0 legacy=0x7f5aa814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:49:42.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.071+0000 7f5aacfa3640 1 --1- 172.21.15.71:0/3003024874 >> v1:172.21.15.71:6790/0 conn(0x7f5aa81430c0 0x7f5aa814dc30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:49:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.074+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361941233 0 0) 0x7f5aa8142520 con 0x7f5aa813f710 2024-03-25T19:49:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.074+0000 7f5aa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.074+0000 7f5aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.074+0000 7f5aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.074+0000 7f5aa6ffd640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:49:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.074+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a90003520 con 0x7f5aa813f710 2024-03-25T19:49:42.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.095+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702649286 0 0) 0x7f5aa80b28a0 con 0x7f5aa80a98f0 2024-03-25T19:49:42.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.095+0000 7f5aa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:42.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.095+0000 7f5aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:42.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.095+0000 7f5aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:42.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.095+0000 7f5aa6ffd640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:49:42.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.095+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa8142520 con 0x7f5aa80a98f0 2024-03-25T19:49:42.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 511592252 0 0) 0x7f5a90003520 con 0x7f5aa813f710 2024-03-25T19:49:42.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 >> v1:172.21.15.71:6790/0 conn(0x7f5aa81430c0 legacy=0x7f5aa814dc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:49:42.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 >> v1:172.21.15.193:6789/0 conn(0x7f5aa80a98f0 legacy=0x7f5aa80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa80b44c0 con 0x7f5aa813f710 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5a98001690 con 0x7f5aa813f710 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aaea06640 5 monclient: authenticate success, global_id 4635 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aaea06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:42.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aa6ffd640 10 monclient: dump: 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:42.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.097+0000 7f5aaea06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:42.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a980017f0 con 0x7f5aa813f710 2024-03-25T19:49:42.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aa6ffd640 1 -- 172.21.15.71:0/3003024874 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5a98001c00 con 0x7f5aa813f710 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aaea06640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aaea06640 4 set_mon_vals no callback set 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aa6ffd640 10 monclient: dump: 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:42.105 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:42.106 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aaea06640 10 monclient: shutdown 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aaea06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.105+0000 7f5aaea06640 1 -- 172.21.15.71:0/3003024874 >> v1:172.21.15.71:6789/0 conn(0x7f5aa813f710 legacy=0x7f5aa81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.106+0000 7f5aaea06640 1 -- 172.21.15.71:0/3003024874 shutdown_connections 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.106+0000 7f5aaea06640 1 -- 172.21.15.71:0/3003024874 >> 172.21.15.71:0/3003024874 conn(0x7f5aa80b7ea0 msgr2=0x7f5aa80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.106+0000 7f5aaea06640 1 -- 172.21.15.71:0/3003024874 shutdown_connections 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.106+0000 7f5aaea06640 1 -- 172.21.15.71:0/3003024874 wait complete. 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.106+0000 7f5aaea06640 10 monclient: build_initial_monmap 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.106+0000 7f5aaea06640 10 monclient: monmap: 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:42.107 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.107+0000 7f5aaea06640 1 librados: starting msgr at 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.107+0000 7f5aaea06640 1 librados: starting objecter 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.107+0000 7f5aaea06640 1 Processor -- start 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.107+0000 7f5aaea06640 1 -- start start 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.107+0000 7f5aaea06640 1 librados: setting wanted keys 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.107+0000 7f5aaea06640 1 librados: calling monclient init 2024-03-25T19:49:42.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.107+0000 7f5aaea06640 10 monclient: init 2024-03-25T19:49:42.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:42.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:42.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 10 monclient(hunting): picked mon.noname-c con 0x7f5aa80a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:42.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 10 monclient(hunting): picked mon.noname-b con 0x7f5aa813f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:42.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 10 monclient(hunting): picked mon.noname-a con 0x7f5aa81430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:42.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa81d7df0 con 0x7f5aa81430c0 2024-03-25T19:49:42.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa81db1e0 con 0x7f5aa813f710 2024-03-25T19:49:42.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa81336f0 con 0x7f5aa80a98f0 2024-03-25T19:49:42.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:42.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.108+0000 7f5aaea06640 10 monclient(hunting): authenticate will time out at 5388.420064s 2024-03-25T19:49:42.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.118+0000 7f5aa7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5aa80a98f0 0x7f5aa81d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56238/0 (socket says 172.21.15.71:56238) 2024-03-25T19:49:42.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.118+0000 7f5aa7fff640 1 -- 172.21.15.71:0/310073510 learned_addr learned my addr 172.21.15.71:0/310073510 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:42.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.119+0000 7f5aa4ff9640 10 client.?.objecter ms_handle_connect 0x7f5aa80a98f0 2024-03-25T19:49:42.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.119+0000 7f5aa4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:42.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.121+0000 7f5aa4ff9640 10 client.?.objecter ms_handle_connect 0x7f5aa813f710 2024-03-25T19:49:42.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.121+0000 7f5aa4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:42.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.132+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926897297 0 0) 0x7f5aa81db1e0 con 0x7f5aa813f710 2024-03-25T19:49:42.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.132+0000 7f5aa4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:42.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.132+0000 7f5aa4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:42.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.132+0000 7f5aa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:42.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.132+0000 7f5aa4ff9640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:49:42.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.132+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a94004eb0 con 0x7f5aa813f710 2024-03-25T19:49:42.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.136+0000 7f5aa4ff9640 10 client.?.objecter ms_handle_connect 0x7f5aa81430c0 2024-03-25T19:49:42.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.136+0000 7f5aa4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:42.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.142+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530391586 0 0) 0x7f5aa81336f0 con 0x7f5aa80a98f0 2024-03-25T19:49:42.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.142+0000 7f5aa4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:42.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.142+0000 7f5aa4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:42.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.142+0000 7f5aa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:42.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.142+0000 7f5aa4ff9640 10 monclient(hunting): my global_id is 14630 2024-03-25T19:49:42.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.142+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa81db1e0 con 0x7f5aa80a98f0 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.143+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1245827586 0 0) 0x7f5a94004eb0 con 0x7f5aa813f710 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.143+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aa81336f0 con 0x7f5aa813f710 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.143+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5a98001b00 con 0x7f5aa813f710 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.143+0000 7f5aa4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.143+0000 7f5aa4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.143+0000 7f5aa4ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:42.143 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:42.144 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:42.144 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:42.144 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:42.144 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:42.144 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:42.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.143+0000 7f5aa4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:42.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.146+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037039058 0 0) 0x7f5aa81d7df0 con 0x7f5aa81430c0 2024-03-25T19:49:42.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.146+0000 7f5aa4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:42.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.146+0000 7f5aa4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:42.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.146+0000 7f5aa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:42.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.146+0000 7f5aa4ff9640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:49:42.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.146+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a94004eb0 con 0x7f5aa81430c0 2024-03-25T19:49:42.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 935943347 0 0) 0x7f5aa81336f0 con 0x7f5aa813f710 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 >> v1:172.21.15.71:6790/0 conn(0x7f5aa80a98f0 legacy=0x7f5aa81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 >> v1:172.21.15.71:6789/0 conn(0x7f5aa81430c0 legacy=0x7f5aa81e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 1 monclient: found mon.b 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8133880 con 0x7f5aa813f710 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aa4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 10 monclient: _renew_subs 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:42.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa81dabe0 con 0x7f5aa813f710 2024-03-25T19:49:42.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:42.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 10 monclient: _renew_subs 2024-03-25T19:49:42.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:42.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa813b730 con 0x7f5aa813f710 2024-03-25T19:49:42.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.147+0000 7f5aaea06640 1 librados: init done 2024-03-25T19:49:42.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.148+0000 7f5aaea06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:42.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.148+0000 7f5aaea06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:42.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.148+0000 7f5aaea06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:42.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.148+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa81db370 con 0x7f5aa813f710 2024-03-25T19:49:42.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.430+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a98001220 con 0x7f5aa813f710 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.430+0000 7f5aa4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.430+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5a98004c30 con 0x7f5aa813f710 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.430+0000 7f5aa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.430+0000 7f5aa4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.430+0000 7f5aa4ff9640 10 monclient: dump: 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:42.430 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:42.431 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:42.431 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:42.431 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:42.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.430+0000 7f5aa67fc640 4 set_mon_vals no callback set 2024-03-25T19:49:42.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.430+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f5a9801ac40 con 0x7f5aa813f710 2024-03-25T19:49:42.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.431+0000 7f5aa7fff640 10 monclient: get_auth_request con 0x7f5a9406de00 auth_method 0 2024-03-25T19:49:42.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.431+0000 7f5aa4ff9640 10 client.4636.objecter ms_handle_connect 0x7f5a9406de00 2024-03-25T19:49:42.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.434+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f5a98054840 con 0x7f5aa813f710 2024-03-25T19:49:42.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.434+0000 7f5aa4ff9640 10 client.4636.objecter ms_dispatch 0x7f5aa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:42.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.434+0000 7f5aa4ff9640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:42.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.434+0000 7f5aa4ff9640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:42.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.434+0000 7f5aa4ff9640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T19:49:42.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.437+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a98050180 con 0x7f5aa813f710 2024-03-25T19:49:42.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.437+0000 7f5aa4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:42.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.437+0000 7f5aa4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:42.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.690+0000 7f5aaea06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:42.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.690+0000 7f5aaea06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:42.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.690+0000 7f5aaea06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:42.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.690+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aa814f2d0 con 0x7f5aa813f710 2024-03-25T19:49:42.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.724+0000 7f5aa4ff9640 1 -- 172.21.15.71:0/310073510 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f5a98052280 con 0x7f5aa813f710 2024-03-25T19:49:42.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.724+0000 7f5aa4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:42.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.724+0000 7f5aa4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:42.724 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:49:42.724 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:42.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 10 librados: watch_flush enter 2024-03-25T19:49:42.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 10 librados: watch_flush exit 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 10 client.4636.objecter successfully canceled tick 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 >> v1:172.21.15.71:6800/3911735624 conn(0x7f5a9406de00 legacy=0x7f5a940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 10 monclient: shutdown 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 >> v1:172.21.15.193:6789/0 conn(0x7f5aa813f710 legacy=0x7f5aa8132fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 shutdown_connections 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 >> 172.21.15.71:0/310073510 conn(0x7f5aa80b7ea0 msgr2=0x7f5aa8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.726+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 shutdown_connections 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.727+0000 7f5aaea06640 1 -- 172.21.15.71:0/310073510 wait complete. 2024-03-25T19:49:42.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:42.727+0000 7f5aaea06640 1 librados: shutdown 2024-03-25T19:49:42.739 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:47.960 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:30.716 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:47.971 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:30.727 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:48.012 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:30.768 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:48.019 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:30.738 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:48.022 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:30.778 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:48.025 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:30.744 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:48.034 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:30.791 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:48.049 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:30.768 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:48.063 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:30.819 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:49.545 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:49.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.659+0000 7f080aee0640 10 monclient: get_monmap_and_config 2024-03-25T19:49:49.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.659+0000 7f080aee0640 10 monclient: build_initial_monmap 2024-03-25T19:49:49.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.659+0000 7f080aee0640 10 monclient: monmap: 2024-03-25T19:49:49.659 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:49.659 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:49.659 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:49:49.660229+0000 2024-03-25T19:49:49.659 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:49:49.660229+0000 2024-03-25T19:49:49.660 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:49.660 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:49.660 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:49.660 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:49:49.660 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:49:49.660 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:49.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.660+0000 7f080aee0640 1 Processor -- start 2024-03-25T19:49:49.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.660+0000 7f080aee0640 1 -- start start 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.660+0000 7f080aee0640 10 monclient: init 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 10 monclient(hunting): picked mon.noname-a con 0x7f08040a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 10 monclient(hunting): picked mon.noname-c con 0x7f080413f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 10 monclient(hunting): picked mon.noname-b con 0x7f08041430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0804142520 con 0x7f08040a98f0 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08040b28a0 con 0x7f080413f710 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08040b2a30 con 0x7f08041430c0 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:49.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.661+0000 7f080aee0640 10 monclient(hunting): authenticate will time out at 5395.973014s 2024-03-25T19:49:49.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.673+0000 7f0803fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f080413f710 0x7f08041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:50160/0 (socket says 172.21.15.71:50160) 2024-03-25T19:49:49.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.673+0000 7f0803fff640 1 -- 172.21.15.71:0/3554067042 learned_addr learned my addr 172.21.15.71:0/3554067042 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:49.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.686+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013020328 0 0) 0x7f08040b28a0 con 0x7f080413f710 2024-03-25T19:49:49.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.686+0000 7f08037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:49.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.686+0000 7f08037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:49.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.686+0000 7f08037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:49.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.686+0000 7f08037fe640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:49:49.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.686+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f4003520 con 0x7f080413f710 2024-03-25T19:49:49.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.687+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187155336 0 0) 0x7f08040b2a30 con 0x7f08041430c0 2024-03-25T19:49:49.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.687+0000 7f08037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:49.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.687+0000 7f08037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:49.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.687+0000 7f08037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:49.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.687+0000 7f08037fe640 10 monclient(hunting): my global_id is 14633 2024-03-25T19:49:49.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.687+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08040b28a0 con 0x7f08041430c0 2024-03-25T19:49:49.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 579442371 0 0) 0x7f08040b28a0 con 0x7f08041430c0 2024-03-25T19:49:49.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 >> v1:172.21.15.193:6789/0 conn(0x7f080413f710 legacy=0x7f08041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:49.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 >> v1:172.21.15.71:6789/0 conn(0x7f08040a98f0 legacy=0x7f08040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:49.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:49.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 1 monclient: found mon.noname-b 2024-03-25T19:49:49.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:49.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08040b44c0 con 0x7f08041430c0 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07fc0019f0 con 0x7f08041430c0 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f08037fe640 10 monclient: dump: 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:49.705 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:49.706 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:49.706 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:49.706 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:49.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f080aee0640 5 monclient: authenticate success, global_id 14633 2024-03-25T19:49:49.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.704+0000 7f080aee0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.715+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07fc001b50 con 0x7f08041430c0 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.715+0000 7f08037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.715+0000 7f08037fe640 1 -- 172.21.15.71:0/3554067042 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07fc001f20 con 0x7f08041430c0 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.715+0000 7f08037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.715+0000 7f08037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.715+0000 7f08037fe640 10 monclient: dump: 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:49.715 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.715+0000 7f080aee0640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:49.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.715+0000 7f080aee0640 4 set_mon_vals no callback set 2024-03-25T19:49:49.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.716+0000 7f080aee0640 10 monclient: shutdown 2024-03-25T19:49:49.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.716+0000 7f080aee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:49.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.716+0000 7f080aee0640 1 -- 172.21.15.71:0/3554067042 >> v1:172.21.15.71:6790/0 conn(0x7f08041430c0 legacy=0x7f080414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.716+0000 7f080aee0640 1 -- 172.21.15.71:0/3554067042 shutdown_connections 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.716+0000 7f080aee0640 1 -- 172.21.15.71:0/3554067042 >> 172.21.15.71:0/3554067042 conn(0x7f08040b7ea0 msgr2=0x7f08040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.716+0000 7f080aee0640 1 -- 172.21.15.71:0/3554067042 shutdown_connections 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.716+0000 7f080aee0640 1 -- 172.21.15.71:0/3554067042 wait complete. 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 10 monclient: build_initial_monmap 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 10 monclient: monmap: 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:49:49.719 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 1 librados: starting msgr at 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 1 librados: starting objecter 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 1 Processor -- start 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 1 -- start start 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 1 librados: setting wanted keys 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 1 librados: calling monclient init 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.717+0000 7f080aee0640 10 monclient: init 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:49.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 10 monclient(hunting): picked mon.noname-c con 0x7f08040a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:49.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 10 monclient(hunting): picked mon.noname-a con 0x7f080413f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:49.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 10 monclient(hunting): picked mon.noname-b con 0x7f08041430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:49.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0804137e20 con 0x7f080413f710 2024-03-25T19:49:49.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08040b1720 con 0x7f08041430c0 2024-03-25T19:49:49.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0804138710 con 0x7f08040a98f0 2024-03-25T19:49:49.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:49.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.718+0000 7f080aee0640 10 monclient(hunting): authenticate will time out at 5396.030792s 2024-03-25T19:49:49.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.725+0000 7f080947d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f08041430c0 0x7f08041daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:50204/0 (socket says 172.21.15.71:50204) 2024-03-25T19:49:49.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.725+0000 7f080947d640 1 -- 172.21.15.71:0/2231404865 learned_addr learned my addr 172.21.15.71:0/2231404865 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:49.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.730+0000 7f08017fa640 10 client.?.objecter ms_handle_connect 0x7f08041430c0 2024-03-25T19:49:49.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.730+0000 7f08017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:49.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.736+0000 7f08017fa640 10 client.?.objecter ms_handle_connect 0x7f08040a98f0 2024-03-25T19:49:49.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.736+0000 7f08017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:49.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.736+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461423912 0 0) 0x7f08040b1720 con 0x7f08041430c0 2024-03-25T19:49:49.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.736+0000 7f08017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:49.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.736+0000 7f08017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:49.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.736+0000 7f08017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:49.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.736+0000 7f08017fa640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:49:49.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.736+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e8004eb0 con 0x7f08041430c0 2024-03-25T19:49:49.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.738+0000 7f08017fa640 10 client.?.objecter ms_handle_connect 0x7f080413f710 2024-03-25T19:49:49.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.738+0000 7f08017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.747+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4269739901 0 0) 0x7f07e8004eb0 con 0x7f08041430c0 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.747+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08040b1720 con 0x7f08041430c0 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.747+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07fc001f60 con 0x7f08041430c0 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.747+0000 7f08017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.747+0000 7f08017fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.747+0000 7f08017fa640 10 monclient(hunting): dump: 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:49.747 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:49.748 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:49.748 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:49.748 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:49.748 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:49.748 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:49.748 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:49.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.747+0000 7f08017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828892569 0 0) 0x7f0804137e20 con 0x7f080413f710 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e8004eb0 con 0x7f080413f710 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409362221 0 0) 0x7f0804138710 con 0x7f08040a98f0 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:49.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 10 monclient(hunting): my global_id is 14636 2024-03-25T19:49:49.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.748+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0804137e20 con 0x7f08040a98f0 2024-03-25T19:49:49.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3312484582 0 0) 0x7f08040b1720 con 0x7f08041430c0 2024-03-25T19:49:49.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 >> v1:172.21.15.71:6790/0 conn(0x7f08040a98f0 legacy=0x7f0804137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:49.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 >> v1:172.21.15.71:6789/0 conn(0x7f080413f710 legacy=0x7f08041da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 10 monclient: _finish_hunting 0 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 1 monclient: found mon.b 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0804138040 con 0x7f08041430c0 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 10 monclient: _finish_auth 0 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 10 monclient: _check_auth_tickets 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f08017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 5 monclient: authenticate success, global_id 4642 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 10 monclient: _renew_subs 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08041382f0 con 0x7f08041430c0 2024-03-25T19:49:49.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:49.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 10 monclient: _renew_subs 2024-03-25T19:49:49.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:49.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08041e2fb0 con 0x7f08041430c0 2024-03-25T19:49:49.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.758+0000 7f080aee0640 1 librados: init done 2024-03-25T19:49:49.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.760+0000 7f080aee0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:49.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.760+0000 7f080aee0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:49.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.760+0000 7f080aee0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:49.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:49.760+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08040b3ef0 con 0x7f08041430c0 2024-03-25T19:49:50.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.037+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07fc003c20 con 0x7f08041430c0 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.037+0000 7f08017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.037+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f07fc003fb0 con 0x7f08041430c0 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.037+0000 7f08017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.037+0000 7f08017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.037+0000 7f0802ffd640 4 set_mon_vals no callback set 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.037+0000 7f08017fa640 10 monclient: dump: 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:50.038 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:50.039 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:50.039 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:50.039 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:50.039 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:50.039 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:50.039 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:50.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.038+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f07fc0041d0 con 0x7f08041430c0 2024-03-25T19:49:50.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.038+0000 7f0808c7c640 10 monclient: get_auth_request con 0x7f07e806de30 auth_method 0 2024-03-25T19:49:50.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.039+0000 7f08017fa640 10 client.4642.objecter ms_handle_connect 0x7f07e806de30 2024-03-25T19:49:50.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.043+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f07fc054be0 con 0x7f08041430c0 2024-03-25T19:49:50.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.043+0000 7f08017fa640 10 client.4642.objecter ms_dispatch 0x7f0804000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:50.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.043+0000 7f08017fa640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:50.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.043+0000 7f08017fa640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:50.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.043+0000 7f08017fa640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T19:49:50.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.043+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07fc0506f0 con 0x7f08041430c0 2024-03-25T19:49:50.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.043+0000 7f08017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:50.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.043+0000 7f08017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:50.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.288+0000 7f080aee0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:50.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.288+0000 7f080aee0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:50.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.288+0000 7f080aee0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:50.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.288+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08040627b0 con 0x7f08041430c0 2024-03-25T19:49:50.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.317+0000 7f08017fa640 1 -- 172.21.15.71:0/2231404865 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f07fc052620 con 0x7f08041430c0 2024-03-25T19:49:50.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.317+0000 7f08017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:50.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.317+0000 7f08017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:50.317 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:49:50.317 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:50.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 10 librados: watch_flush enter 2024-03-25T19:49:50.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 10 librados: watch_flush exit 2024-03-25T19:49:50.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T19:49:50.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 10 client.4642.objecter successfully canceled tick 2024-03-25T19:49:50.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 >> v1:172.21.15.71:6800/3911735624 conn(0x7f07e806de30 legacy=0x7f07e80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:50.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 10 monclient: shutdown 2024-03-25T19:49:50.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:50.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 >> v1:172.21.15.193:6789/0 conn(0x7f08041430c0 legacy=0x7f08041daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:50.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 shutdown_connections 2024-03-25T19:49:50.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 >> 172.21.15.71:0/2231404865 conn(0x7f08040b7ea0 msgr2=0x7f0804074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:50.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 shutdown_connections 2024-03-25T19:49:50.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 1 -- 172.21.15.71:0/2231404865 wait complete. 2024-03-25T19:49:50.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:50.320+0000 7f080aee0640 1 librados: shutdown 2024-03-25T19:49:50.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:57.140 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.251+0000 7f66e6d1c640 10 monclient: get_monmap_and_config 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.251+0000 7f66e6d1c640 10 monclient: build_initial_monmap 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.251+0000 7f66e6d1c640 10 monclient: monmap: 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:49:57.252368+0000 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:49:57.252368+0000 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:57.251 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:57.252 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:57.252 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:49:57.252 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:49:57.252 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.252+0000 7f66e6d1c640 1 Processor -- start 2024-03-25T19:49:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.252+0000 7f66e6d1c640 1 -- start start 2024-03-25T19:49:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.252+0000 7f66e6d1c640 10 monclient: init 2024-03-25T19:49:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.252+0000 7f66e6d1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.252+0000 7f66e6d1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.252+0000 7f66e6d1c640 10 monclient(hunting): picked mon.noname-a con 0x7f66e00b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.252+0000 7f66e6d1c640 10 monclient(hunting): picked mon.noname-b con 0x7f66e00b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.253+0000 7f66e6d1c640 10 monclient(hunting): picked mon.noname-c con 0x7f66e014be90 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.253+0000 7f66e6d1c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e00af940 con 0x7f66e00b4d60 2024-03-25T19:49:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.253+0000 7f66e6d1c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e00b0a70 con 0x7f66e014be90 2024-03-25T19:49:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.253+0000 7f66e6d1c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e00b0d50 con 0x7f66e00b1730 2024-03-25T19:49:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.253+0000 7f66e6d1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.253+0000 7f66e6d1c640 10 monclient(hunting): authenticate will time out at 5403.564943s 2024-03-25T19:49:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.261+0000 7f66e52b9640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f66e014be90 0x7f66e014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:58628/0 (socket says 172.21.15.71:58628) 2024-03-25T19:49:57.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.261+0000 7f66e52b9640 1 -- 172.21.15.71:0/1767865809 learned_addr learned my addr 172.21.15.71:0/1767865809 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:57.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.276+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681421799 0 0) 0x7f66e00b0a70 con 0x7f66e014be90 2024-03-25T19:49:57.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.276+0000 7f66df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:57.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.276+0000 7f66df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:57.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.276+0000 7f66df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:57.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.276+0000 7f66df7fe640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:49:57.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.276+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d0003520 con 0x7f66e014be90 2024-03-25T19:49:57.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.279+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953335555 0 0) 0x7f66e00af940 con 0x7f66e00b4d60 2024-03-25T19:49:57.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.279+0000 7f66df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:57.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.279+0000 7f66df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:57.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.279+0000 7f66df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:57.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.279+0000 7f66df7fe640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:49:57.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.279+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e00b0a70 con 0x7f66e00b4d60 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636468091 0 0) 0x7f66d0003520 con 0x7f66e014be90 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 >> v1:172.21.15.71:6790/0 conn(0x7f66e00b1730 legacy=0x7f66e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 >> v1:172.21.15.71:6789/0 conn(0x7f66e00b4d60 legacy=0x7f66e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 1 monclient: found mon.noname-c 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e00b1030 con 0x7f66e014be90 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:57.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f66d8001a10 con 0x7f66e014be90 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66df7fe640 10 monclient: dump: 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:57.289 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:57.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66e6d1c640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:49:57.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.288+0000 7f66e6d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.307+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66d8001b70 con 0x7f66e014be90 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.307+0000 7f66df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.307+0000 7f66df7fe640 1 -- 172.21.15.71:0/1767865809 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f66d8001f80 con 0x7f66e014be90 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.307+0000 7f66e6d1c640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.307+0000 7f66e6d1c640 4 set_mon_vals no callback set 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.307+0000 7f66df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.307+0000 7f66df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.307+0000 7f66df7fe640 10 monclient: dump: 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:57.307 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.308 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.308 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:57.308 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:57.308 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:57.308 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:57.308 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:57.308 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 10 monclient: shutdown 2024-03-25T19:49:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 1 -- 172.21.15.71:0/1767865809 >> v1:172.21.15.193:6789/0 conn(0x7f66e014be90 legacy=0x7f66e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 1 -- 172.21.15.71:0/1767865809 shutdown_connections 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 1 -- 172.21.15.71:0/1767865809 >> 172.21.15.71:0/1767865809 conn(0x7f66e00617a0 msgr2=0x7f66e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 1 -- 172.21.15.71:0/1767865809 shutdown_connections 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 1 -- 172.21.15.71:0/1767865809 wait complete. 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 10 monclient: build_initial_monmap 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.308+0000 7f66e6d1c640 10 monclient: monmap: 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:49:57.311 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.309+0000 7f66e6d1c640 1 librados: starting msgr at 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.309+0000 7f66e6d1c640 1 librados: starting objecter 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.309+0000 7f66e6d1c640 1 Processor -- start 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.309+0000 7f66e6d1c640 1 -- start start 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.309+0000 7f66e6d1c640 1 librados: setting wanted keys 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.309+0000 7f66e6d1c640 1 librados: calling monclient init 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.309+0000 7f66e6d1c640 10 monclient: init 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 10 monclient(hunting): picked mon.noname-b con 0x7f66e00b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 10 monclient(hunting): picked mon.noname-a con 0x7f66e00b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 10 monclient(hunting): picked mon.noname-c con 0x7f66e014be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:49:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e0137a30 con 0x7f66e00b4d60 2024-03-25T19:49:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e00a9470 con 0x7f66e00b1730 2024-03-25T19:49:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e00aa5f0 con 0x7f66e014be90 2024-03-25T19:49:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.310+0000 7f66e6d1c640 10 monclient(hunting): authenticate will time out at 5403.622338s 2024-03-25T19:49:57.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.316+0000 7f66e4ab8640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f66e00b1730 0x7f66e0138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:58664/0 (socket says 172.21.15.71:58664) 2024-03-25T19:49:57.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.316+0000 7f66e4ab8640 1 -- 172.21.15.71:0/3593590854 learned_addr learned my addr 172.21.15.71:0/3593590854 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:49:57.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.318+0000 7f66dd7fa640 10 client.?.objecter ms_handle_connect 0x7f66e014be90 2024-03-25T19:49:57.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.318+0000 7f66dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:57.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.324+0000 7f66dd7fa640 10 client.?.objecter ms_handle_connect 0x7f66e00b1730 2024-03-25T19:49:57.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.324+0000 7f66dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.325+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547497375 0 0) 0x7f66e00aa5f0 con 0x7f66e014be90 2024-03-25T19:49:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.325+0000 7f66dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.325+0000 7f66dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.325+0000 7f66dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.325+0000 7f66dd7fa640 10 monclient(hunting): my global_id is 14642 2024-03-25T19:49:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.325+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c4004eb0 con 0x7f66e014be90 2024-03-25T19:49:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.327+0000 7f66dd7fa640 10 client.?.objecter ms_handle_connect 0x7f66e00b4d60 2024-03-25T19:49:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.327+0000 7f66dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.327+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952912062 0 0) 0x7f66e00a9470 con 0x7f66e00b1730 2024-03-25T19:49:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.327+0000 7f66dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.327+0000 7f66dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.327+0000 7f66dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.327+0000 7f66dd7fa640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:49:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.327+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e00aa5f0 con 0x7f66e00b1730 2024-03-25T19:49:57.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.338+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441177090 0 0) 0x7f66e0137a30 con 0x7f66e00b4d60 2024-03-25T19:49:57.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.338+0000 7f66dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:57.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.338+0000 7f66dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:57.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.338+0000 7f66dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:57.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.338+0000 7f66dd7fa640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:49:57.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.338+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e00a9470 con 0x7f66e00b4d60 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.339+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1157698932 0 0) 0x7f66e00aa5f0 con 0x7f66e00b1730 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.339+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66e0137a30 con 0x7f66e00b1730 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.339+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f66d40019f0 con 0x7f66e00b1730 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.339+0000 7f66dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.339+0000 7f66dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.339+0000 7f66dd7fa640 10 monclient(hunting): dump: 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:57.339 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.340 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.340 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:57.340 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:57.340 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:57.340 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:57.340 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:57.340 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:57.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.339+0000 7f66dd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:57.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.347+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3640495872 0 0) 0x7f66c4004eb0 con 0x7f66e014be90 2024-03-25T19:49:57.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.347+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66e00aa5f0 con 0x7f66e014be90 2024-03-25T19:49:57.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.347+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f66d8001e80 con 0x7f66e014be90 2024-03-25T19:49:57.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.347+0000 7f66dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:57.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.347+0000 7f66dd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:57.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.347+0000 7f66dd7fa640 10 monclient(hunting): dump: 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:57.348 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:57.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3273442691 0 0) 0x7f66e0137a30 con 0x7f66e00b1730 2024-03-25T19:49:57.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 >> v1:172.21.15.71:6790/0 conn(0x7f66e014be90 legacy=0x7f66e0137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:57.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 >> v1:172.21.15.71:6789/0 conn(0x7f66e00b4d60 legacy=0x7f66e0136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:57.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:49:57.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 1 monclient: found mon.b 2024-03-25T19:49:57.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e00aa8d0 con 0x7f66e00b1730 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 10 monclient: _renew_subs 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e00aabb0 con 0x7f66e00b1730 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 10 monclient: _renew_subs 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:57.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e00aa180 con 0x7f66e00b1730 2024-03-25T19:49:57.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.352+0000 7f66e6d1c640 1 librados: init done 2024-03-25T19:49:57.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.353+0000 7f66e6d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:57.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.353+0000 7f66e6d1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:57.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.353+0000 7f66e6d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:57.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.353+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e00627b0 con 0x7f66e00b1730 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.572+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66d4003610 con 0x7f66e00b1730 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.572+0000 7f66dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.572+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f66d40039a0 con 0x7f66e00b1730 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.572+0000 7f66dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.572+0000 7f66dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.572+0000 7f66dd7fa640 10 monclient: dump: 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:49:57.572 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:57.573 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:49:57.573 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:49:57.573 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:49:57.573 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:49:57.573 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:49:57.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.572+0000 7f66deffd640 4 set_mon_vals no callback set 2024-03-25T19:49:57.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.626+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f66d4003b00 con 0x7f66e00b1730 2024-03-25T19:49:57.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.626+0000 7f66dffff640 10 monclient: get_auth_request con 0x7f66c406df70 auth_method 0 2024-03-25T19:49:57.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.627+0000 7f66dd7fa640 10 client.4648.objecter ms_handle_connect 0x7f66c406df70 2024-03-25T19:49:57.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.629+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f66d40547f0 con 0x7f66e00b1730 2024-03-25T19:49:57.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.629+0000 7f66dd7fa640 10 client.4648.objecter ms_dispatch 0x7f66e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:57.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.629+0000 7f66dd7fa640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:57.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.629+0000 7f66dd7fa640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:57.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.629+0000 7f66dd7fa640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T19:49:57.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.632+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66d4050300 con 0x7f66e00b1730 2024-03-25T19:49:57.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.632+0000 7f66dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:57.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.632+0000 7f66dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:57.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.884+0000 7f66e6d1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:57.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.884+0000 7f66e6d1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:57.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.884+0000 7f66e6d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:49:57.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.884+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66e0069f40 con 0x7f66e00b1730 2024-03-25T19:49:57.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.908+0000 7f66dd7fa640 1 -- 172.21.15.71:0/3593590854 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f66d4052230 con 0x7f66e00b1730 2024-03-25T19:49:57.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.908+0000 7f66dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:57.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.908+0000 7f66dd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:57.908 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:49:57.908 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:57.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 10 librados: watch_flush enter 2024-03-25T19:49:57.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 10 librados: watch_flush exit 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 10 client.4648.objecter successfully canceled tick 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 >> v1:172.21.15.71:6800/3911735624 conn(0x7f66c406df70 legacy=0x7f66c4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 10 monclient: shutdown 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 >> v1:172.21.15.193:6789/0 conn(0x7f66e00b1730 legacy=0x7f66e0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 shutdown_connections 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 >> 172.21.15.71:0/3593590854 conn(0x7f66e00617a0 msgr2=0x7f66e0145100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 shutdown_connections 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 1 -- 172.21.15.71:0/3593590854 wait complete. 2024-03-25T19:49:57.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:49:57.910+0000 7f66e6d1c640 1 librados: shutdown 2024-03-25T19:49:57.922 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:04.728 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.825+0000 7fd35c84e640 10 monclient: get_monmap_and_config 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.825+0000 7fd35c84e640 10 monclient: build_initial_monmap 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.825+0000 7fd35c84e640 10 monclient: monmap: 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:50:04.825934+0000 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:50:04.825934+0000 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:04.825 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:04.826 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:04.826 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:50:04.826 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:50:04.826 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:04.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.825+0000 7fd35c84e640 1 Processor -- start 2024-03-25T19:50:04.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.825+0000 7fd35c84e640 1 -- start start 2024-03-25T19:50:04.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.825+0000 7fd35c84e640 10 monclient: init 2024-03-25T19:50:04.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:04.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:04.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3580ae6e0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:04.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 10 monclient(hunting): picked mon.noname-b con 0x7fd35806f080 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:04.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3580704e0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:04.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3580718d0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35806fa30 con 0x7fd3580704e0 2024-03-25T19:50:04.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd358070090 con 0x7fd35806f080 2024-03-25T19:50:04.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:04.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.826+0000 7fd35c84e640 10 monclient(hunting): authenticate will time out at 5411.138402s 2024-03-25T19:50:04.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.836+0000 7fd35659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd3580ae6e0 0x7fd35813c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33732/0 (socket says 172.21.15.71:33732) 2024-03-25T19:50:04.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.836+0000 7fd35659c640 1 -- 172.21.15.71:0/1326788171 learned_addr learned my addr 172.21.15.71:0/1326788171 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:04.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.848+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829206206 0 0) 0x7fd3580718d0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.848+0000 7fd35559a640 10 monclient(hunting): _negotiate 2024-03-25T19:50:04.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.848+0000 7fd35559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:04.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.848+0000 7fd35559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:04.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.848+0000 7fd35559a640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:50:04.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.848+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340003520 con 0x7fd3580ae6e0 2024-03-25T19:50:04.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 565835453 0 0) 0x7fd340003520 con 0x7fd3580ae6e0 2024-03-25T19:50:04.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 >> v1:172.21.15.71:6790/0 conn(0x7fd35806f080 legacy=0x7fd358071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 >> v1:172.21.15.193:6789/0 conn(0x7fd3580704e0 legacy=0x7fd35814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 10 monclient: _finish_hunting 0 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 1 monclient: found mon.noname-a 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35806f510 con 0x7fd3580ae6e0 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 10 monclient: _finish_auth 0 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 10 monclient: _check_auth_tickets 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd348001690 con 0x7fd3580ae6e0 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35559a640 10 monclient: dump: 2024-03-25T19:50:04.884 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35c84e640 5 monclient: authenticate success, global_id 4653 2024-03-25T19:50:04.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.883+0000 7fd35c84e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:04.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.906+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3480017f0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.906+0000 7fd35559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:04.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.906+0000 7fd35559a640 1 -- 172.21.15.71:0/1326788171 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd348001c40 con 0x7fd3580ae6e0 2024-03-25T19:50:04.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.906+0000 7fd35559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:04.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.906+0000 7fd35559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:04.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.906+0000 7fd35559a640 10 monclient: dump: 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.906+0000 7fd35c84e640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:04.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.906+0000 7fd35c84e640 4 set_mon_vals no callback set 2024-03-25T19:50:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.907+0000 7fd35c84e640 10 monclient: shutdown 2024-03-25T19:50:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.907+0000 7fd35c84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.907+0000 7fd35c84e640 1 -- 172.21.15.71:0/1326788171 >> v1:172.21.15.71:6789/0 conn(0x7fd3580ae6e0 legacy=0x7fd35813c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.907+0000 7fd35c84e640 1 -- 172.21.15.71:0/1326788171 shutdown_connections 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.907+0000 7fd35c84e640 1 -- 172.21.15.71:0/1326788171 >> 172.21.15.71:0/1326788171 conn(0x7fd35806b630 msgr2=0x7fd3580b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.907+0000 7fd35c84e640 1 -- 172.21.15.71:0/1326788171 shutdown_connections 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.907+0000 7fd35c84e640 1 -- 172.21.15.71:0/1326788171 wait complete. 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.908+0000 7fd35c84e640 10 monclient: build_initial_monmap 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.908+0000 7fd35c84e640 10 monclient: monmap: 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:04.910 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 1 librados: starting msgr at 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 1 librados: starting objecter 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 1 Processor -- start 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 1 -- start start 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 1 librados: setting wanted keys 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 1 librados: calling monclient init 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 10 monclient: init 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3580704e0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3580ae6e0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 10 monclient(hunting): picked mon.noname-c con 0x7fd358141d10 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:04.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3581421f0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.909+0000 7fd35c84e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd358142710 con 0x7fd3580704e0 2024-03-25T19:50:04.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.910+0000 7fd35c84e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd358135340 con 0x7fd358141d10 2024-03-25T19:50:04.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.910+0000 7fd35c84e640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:04.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.910+0000 7fd35c84e640 10 monclient(hunting): authenticate will time out at 5411.221866s 2024-03-25T19:50:04.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.926+0000 7fd355d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd3580ae6e0 0x7fd3581400b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33802/0 (socket says 172.21.15.71:33802) 2024-03-25T19:50:04.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.926+0000 7fd355d9b640 1 -- 172.21.15.71:0/1784512935 learned_addr learned my addr 172.21.15.71:0/1784512935 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:04.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.926+0000 7fd346ffd640 10 client.?.objecter ms_handle_connect 0x7fd3580ae6e0 2024-03-25T19:50:04.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.926+0000 7fd346ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:04.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.932+0000 7fd346ffd640 10 client.?.objecter ms_handle_connect 0x7fd358141d10 2024-03-25T19:50:04.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.932+0000 7fd346ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:04.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.934+0000 7fd346ffd640 10 client.?.objecter ms_handle_connect 0x7fd3580704e0 2024-03-25T19:50:04.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.934+0000 7fd346ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:04.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.935+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144830758 0 0) 0x7fd3581421f0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.935+0000 7fd346ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:04.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.935+0000 7fd346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:04.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.935+0000 7fd346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:04.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.935+0000 7fd346ffd640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:50:04.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.935+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338004eb0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.951+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 183037072 0 0) 0x7fd338004eb0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.951+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3581421f0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.951+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd34c001a00 con 0x7fd3580ae6e0 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.951+0000 7fd346ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.951+0000 7fd346ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.951+0000 7fd346ffd640 10 monclient(hunting): dump: 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:04.951 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.951+0000 7fd346ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.953+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745146918 0 0) 0x7fd358142710 con 0x7fd3580704e0 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.953+0000 7fd346ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.953+0000 7fd346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:04.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.953+0000 7fd346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:04.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.953+0000 7fd346ffd640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:50:04.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.953+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338004eb0 con 0x7fd3580704e0 2024-03-25T19:50:04.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.956+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403911226 0 0) 0x7fd358135340 con 0x7fd358141d10 2024-03-25T19:50:04.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.956+0000 7fd346ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:04.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.956+0000 7fd346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:04.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.956+0000 7fd346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:04.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.956+0000 7fd346ffd640 10 monclient(hunting): my global_id is 14648 2024-03-25T19:50:04.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.956+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd358142710 con 0x7fd358141d10 2024-03-25T19:50:04.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2709766364 0 0) 0x7fd3581421f0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 >> v1:172.21.15.71:6790/0 conn(0x7fd358141d10 legacy=0x7fd3581e3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 >> v1:172.21.15.193:6789/0 conn(0x7fd3580704e0 legacy=0x7fd358134c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 1 monclient: found mon.a 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3581354d0 con 0x7fd3580ae6e0 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 5 monclient: authenticate success, global_id 4656 2024-03-25T19:50:04.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 10 monclient: _renew_subs 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd358135660 con 0x7fd3580ae6e0 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 10 client.4656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 10 monclient: _renew_subs 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35813ec20 con 0x7fd3580ae6e0 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.965+0000 7fd35c84e640 1 librados: init done 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.967+0000 7fd35c84e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:04.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.967+0000 7fd35c84e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:04.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.967+0000 7fd35c84e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:50:04.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:04.967+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3581c86a0 con 0x7fd3580ae6e0 2024-03-25T19:50:05.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.273+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd34c001c80 con 0x7fd3580ae6e0 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.273+0000 7fd346ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.273+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd34c003a60 con 0x7fd3580ae6e0 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.273+0000 7fd346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.273+0000 7fd346ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.273+0000 7fd346ffd640 10 monclient: dump: 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:05.273 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:05.274 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:05.274 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:05.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.273+0000 7fd354d99640 4 set_mon_vals no callback set 2024-03-25T19:50:05.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.273+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fd34c003c80 con 0x7fd3580ae6e0 2024-03-25T19:50:05.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.274+0000 7fd35659c640 10 monclient: get_auth_request con 0x7fd33806de60 auth_method 0 2024-03-25T19:50:05.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.274+0000 7fd346ffd640 10 client.4656.objecter ms_handle_connect 0x7fd33806de60 2024-03-25T19:50:05.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.280+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fd34c054870 con 0x7fd3580ae6e0 2024-03-25T19:50:05.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.280+0000 7fd346ffd640 10 client.4656.objecter ms_dispatch 0x7fd358000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:05.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.280+0000 7fd346ffd640 3 client.4656.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:05.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.280+0000 7fd346ffd640 3 client.4656.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:05.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.280+0000 7fd346ffd640 20 client.4656.objecter dump_active .. 0 homeless 2024-03-25T19:50:05.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.280+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd34c050380 con 0x7fd3580ae6e0 2024-03-25T19:50:05.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.280+0000 7fd346ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:05.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.280+0000 7fd346ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:05.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.520+0000 7fd35c84e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:05.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.520+0000 7fd35c84e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:05.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.520+0000 7fd35c84e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:50:05.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.520+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd358074d80 con 0x7fd3580ae6e0 2024-03-25T19:50:05.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.554+0000 7fd346ffd640 1 -- 172.21.15.71:0/1784512935 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fd358074d80 con 0x7fd3580ae6e0 2024-03-25T19:50:05.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.554+0000 7fd346ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:05.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.554+0000 7fd346ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:05.554 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:50:05.554 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:05.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 10 librados: watch_flush enter 2024-03-25T19:50:05.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 10 librados: watch_flush exit 2024-03-25T19:50:05.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 20 client.4656.objecter shutdown clearing up homeless session... 2024-03-25T19:50:05.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 10 client.4656.objecter successfully canceled tick 2024-03-25T19:50:05.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 >> v1:172.21.15.71:6800/3911735624 conn(0x7fd33806de60 legacy=0x7fd338070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:05.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 10 monclient: shutdown 2024-03-25T19:50:05.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:05.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 >> v1:172.21.15.71:6789/0 conn(0x7fd3580ae6e0 legacy=0x7fd3581400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:05.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 shutdown_connections 2024-03-25T19:50:05.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 >> 172.21.15.71:0/1784512935 conn(0x7fd35806b630 msgr2=0x7fd358071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:05.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 shutdown_connections 2024-03-25T19:50:05.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 1 -- 172.21.15.71:0/1784512935 wait complete. 2024-03-25T19:50:05.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:05.557+0000 7fd35c84e640 1 librados: shutdown 2024-03-25T19:50:05.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:08.076 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:50.832 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.084 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:50.841 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.119 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:09:50.875 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.130 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:50.849 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.138 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:50.895 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.139 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:50.858 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.145 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:50.901 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.167 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:50.886 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:08.182 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:09:50.939 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:12.376 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:12.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.492+0000 7fb13ac93640 10 monclient: get_monmap_and_config 2024-03-25T19:50:12.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.492+0000 7fb13ac93640 10 monclient: build_initial_monmap 2024-03-25T19:50:12.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.492+0000 7fb13ac93640 10 monclient: monmap: 2024-03-25T19:50:12.492 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:50:12.493704+0000 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:50:12.493704+0000 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:50:12.493 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:12.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.493+0000 7fb13ac93640 1 Processor -- start 2024-03-25T19:50:12.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.493+0000 7fb13ac93640 1 -- start start 2024-03-25T19:50:12.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.493+0000 7fb13ac93640 10 monclient: init 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 10 monclient(hunting): picked mon.noname-c con 0x7fb1340b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 10 monclient(hunting): picked mon.noname-a con 0x7fb1340b1730 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 10 monclient(hunting): picked mon.noname-b con 0x7fb13414be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1340af940 con 0x7fb1340b1730 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1340b0a70 con 0x7fb1340b4d60 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1340b0d50 con 0x7fb13414be90 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:12.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.494+0000 7fb13ac93640 10 monclient(hunting): authenticate will time out at 5418.806581s 2024-03-25T19:50:12.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.507+0000 7fb139230640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb13414be90 0x7fb13414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56590/0 (socket says 172.21.15.71:56590) 2024-03-25T19:50:12.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.507+0000 7fb139230640 1 -- 172.21.15.71:0/658736844 learned_addr learned my addr 172.21.15.71:0/658736844 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:12.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.523+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165468769 0 0) 0x7fb1340b0a70 con 0x7fb1340b4d60 2024-03-25T19:50:12.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.523+0000 7fb1337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:12.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.523+0000 7fb1337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:12.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.523+0000 7fb1337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:12.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.523+0000 7fb1337fe640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:50:12.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.523+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb120003520 con 0x7fb1340b4d60 2024-03-25T19:50:12.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911571238 0 0) 0x7fb1340b0d50 con 0x7fb13414be90 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient(hunting): my global_id is 14651 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1340b0a70 con 0x7fb13414be90 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3122769227 0 0) 0x7fb120003520 con 0x7fb1340b4d60 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 >> v1:172.21.15.71:6790/0 conn(0x7fb13414be90 legacy=0x7fb13414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 >> v1:172.21.15.71:6789/0 conn(0x7fb1340b1730 legacy=0x7fb1340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 1 monclient: found mon.noname-c 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1340b1030 con 0x7fb1340b4d60 2024-03-25T19:50:12.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb11c001690 con 0x7fb1340b4d60 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb1337fe640 10 monclient: dump: 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:12.536 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:12.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb13ac93640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:50:12.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.534+0000 7fb13ac93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:12.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.544+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb11c001870 con 0x7fb1340b4d60 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.544+0000 7fb1337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.544+0000 7fb1337fe640 1 -- 172.21.15.71:0/658736844 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb11c001c80 con 0x7fb1340b4d60 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.544+0000 7fb13ac93640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.544+0000 7fb13ac93640 4 set_mon_vals no callback set 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.544+0000 7fb1337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.544+0000 7fb1337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.544+0000 7fb1337fe640 10 monclient: dump: 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:12.544 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:12.545 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:12.545 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:12.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.545+0000 7fb13ac93640 10 monclient: shutdown 2024-03-25T19:50:12.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.545+0000 7fb13ac93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:12.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.545+0000 7fb13ac93640 1 -- 172.21.15.71:0/658736844 >> v1:172.21.15.193:6789/0 conn(0x7fb1340b4d60 legacy=0x7fb13413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:12.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.545+0000 7fb13ac93640 1 -- 172.21.15.71:0/658736844 shutdown_connections 2024-03-25T19:50:12.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.545+0000 7fb13ac93640 1 -- 172.21.15.71:0/658736844 >> 172.21.15.71:0/658736844 conn(0x7fb1340617a0 msgr2=0x7fb134061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:12.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.545+0000 7fb13ac93640 1 -- 172.21.15.71:0/658736844 shutdown_connections 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.545+0000 7fb13ac93640 1 -- 172.21.15.71:0/658736844 wait complete. 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 10 monclient: build_initial_monmap 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 10 monclient: monmap: 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 1 librados: starting msgr at 2024-03-25T19:50:12.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 1 librados: starting objecter 2024-03-25T19:50:12.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 1 Processor -- start 2024-03-25T19:50:12.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 1 -- start start 2024-03-25T19:50:12.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 1 librados: setting wanted keys 2024-03-25T19:50:12.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 1 librados: calling monclient init 2024-03-25T19:50:12.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.546+0000 7fb13ac93640 10 monclient: init 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.547+0000 7fb13ac93640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.547+0000 7fb13ac93640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.547+0000 7fb13ac93640 10 monclient(hunting): picked mon.noname-c con 0x7fb1340b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.547+0000 7fb13ac93640 10 monclient(hunting): picked mon.noname-b con 0x7fb1340b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.548+0000 7fb13ac93640 10 monclient(hunting): picked mon.noname-a con 0x7fb13414be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.548+0000 7fb13ac93640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb134144150 con 0x7fb13414be90 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.548+0000 7fb13ac93640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb134145840 con 0x7fb1340b4d60 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.548+0000 7fb13ac93640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb134143c30 con 0x7fb1340b1730 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.548+0000 7fb13ac93640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:12.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.548+0000 7fb13ac93640 10 monclient(hunting): authenticate will time out at 5418.859946s 2024-03-25T19:50:12.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.564+0000 7fb133fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fb1340b4d60 0x7fb1341d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:42604/0 (socket says 172.21.15.71:42604) 2024-03-25T19:50:12.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.564+0000 7fb133fff640 1 -- 172.21.15.71:0/3690939008 learned_addr learned my addr 172.21.15.71:0/3690939008 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:12.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.569+0000 7fb1317fa640 10 client.?.objecter ms_handle_connect 0x7fb1340b4d60 2024-03-25T19:50:12.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.569+0000 7fb1317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:12.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.570+0000 7fb1317fa640 10 client.?.objecter ms_handle_connect 0x7fb1340b1730 2024-03-25T19:50:12.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.570+0000 7fb1317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:12.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.575+0000 7fb1317fa640 10 client.?.objecter ms_handle_connect 0x7fb13414be90 2024-03-25T19:50:12.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.575+0000 7fb1317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:12.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.583+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335143819 0 0) 0x7fb134143c30 con 0x7fb1340b1730 2024-03-25T19:50:12.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.583+0000 7fb1317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:12.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.583+0000 7fb1317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:12.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.583+0000 7fb1317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:12.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.583+0000 7fb1317fa640 10 monclient(hunting): my global_id is 14654 2024-03-25T19:50:12.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.583+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb114004eb0 con 0x7fb1340b1730 2024-03-25T19:50:12.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.586+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235938468 0 0) 0x7fb134145840 con 0x7fb1340b4d60 2024-03-25T19:50:12.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.586+0000 7fb1317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:12.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.586+0000 7fb1317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:12.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.586+0000 7fb1317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:12.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.586+0000 7fb1317fa640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:50:12.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.586+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb134143c30 con 0x7fb1340b4d60 2024-03-25T19:50:12.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.592+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196379713 0 0) 0x7fb134144150 con 0x7fb13414be90 2024-03-25T19:50:12.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.592+0000 7fb1317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:12.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.592+0000 7fb1317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:12.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.592+0000 7fb1317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:12.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.592+0000 7fb1317fa640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:50:12.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.592+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb134145840 con 0x7fb13414be90 2024-03-25T19:50:12.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.602+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 174781168 0 0) 0x7fb114004eb0 con 0x7fb1340b1730 2024-03-25T19:50:12.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.602+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb134144150 con 0x7fb1340b1730 2024-03-25T19:50:12.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.602+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb11c001b80 con 0x7fb1340b1730 2024-03-25T19:50:12.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.602+0000 7fb1317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:12.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.602+0000 7fb1317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:12.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.602+0000 7fb1317fa640 10 monclient(hunting): dump: 2024-03-25T19:50:12.602 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:12.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.602+0000 7fb1317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.610+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188234127 0 0) 0x7fb134145840 con 0x7fb13414be90 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.610+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb114004eb0 con 0x7fb13414be90 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.610+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb12c001850 con 0x7fb13414be90 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.610+0000 7fb1317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.610+0000 7fb1317fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.610+0000 7fb1317fa640 10 monclient(hunting): dump: 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:12.610 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:12.611 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:12.611 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:12.611 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3554682152 0 0) 0x7fb134144150 con 0x7fb1340b1730 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 >> v1:172.21.15.193:6789/0 conn(0x7fb1340b4d60 legacy=0x7fb1341d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 >> v1:172.21.15.71:6789/0 conn(0x7fb13414be90 legacy=0x7fb1341e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 10 monclient: _finish_hunting 0 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 1 monclient: found mon.c 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb134143900 con 0x7fb1340b1730 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 10 monclient: _finish_auth 0 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 10 monclient: _check_auth_tickets 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb1317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:12.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb13ac93640 5 monclient: authenticate success, global_id 14654 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb13ac93640 10 monclient: _renew_subs 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb13ac93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.611+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb134137990 con 0x7fb1340b1730 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.612+0000 7fb13ac93640 10 client.14654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.612+0000 7fb13ac93640 10 monclient: _renew_subs 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.612+0000 7fb13ac93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.612+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1341307e0 con 0x7fb1340b1730 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.612+0000 7fb13ac93640 1 librados: init done 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.613+0000 7fb13ac93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:12.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.613+0000 7fb13ac93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:12.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.613+0000 7fb13ac93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:12.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.613+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1341401f0 con 0x7fb1340b1730 2024-03-25T19:50:12.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.907+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb11c001220 con 0x7fb1340b1730 2024-03-25T19:50:12.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.907+0000 7fb1317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:12.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.907+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fb11c003b00 con 0x7fb1340b1730 2024-03-25T19:50:12.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.907+0000 7fb1317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.907+0000 7fb1317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.907+0000 7fb1317fa640 10 monclient: dump: 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:12.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.907+0000 7fb132ffd640 4 set_mon_vals no callback set 2024-03-25T19:50:12.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.908+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fb11c019b10 con 0x7fb1340b1730 2024-03-25T19:50:12.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.909+0000 7fb133fff640 10 monclient: get_auth_request con 0x7fb11406df10 auth_method 0 2024-03-25T19:50:12.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.909+0000 7fb1317fa640 10 client.14654.objecter ms_handle_connect 0x7fb11406df10 2024-03-25T19:50:12.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.914+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fb11c053790 con 0x7fb1340b1730 2024-03-25T19:50:12.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.914+0000 7fb1317fa640 10 client.14654.objecter ms_dispatch 0x7fb134000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:12.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.914+0000 7fb1317fa640 3 client.14654.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:12.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.914+0000 7fb1317fa640 3 client.14654.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:12.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.914+0000 7fb1317fa640 20 client.14654.objecter dump_active .. 0 homeless 2024-03-25T19:50:12.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.914+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb11c04f0d0 con 0x7fb1340b1730 2024-03-25T19:50:12.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.914+0000 7fb1317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:12.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:12.914+0000 7fb1317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:13.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.125+0000 7fb13ac93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:13.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.125+0000 7fb13ac93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:13.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.125+0000 7fb13ac93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:13.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.125+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb134143e20 con 0x7fb1340b1730 2024-03-25T19:50:13.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.192+0000 7fb1317fa640 1 -- 172.21.15.71:0/3690939008 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fb11c0511d0 con 0x7fb1340b1730 2024-03-25T19:50:13.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.192+0000 7fb1317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:13.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.192+0000 7fb1317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:13.193 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:50:13.193 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:13.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 10 librados: watch_flush enter 2024-03-25T19:50:13.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 10 librados: watch_flush exit 2024-03-25T19:50:13.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 20 client.14654.objecter shutdown clearing up homeless session... 2024-03-25T19:50:13.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 10 client.14654.objecter successfully canceled tick 2024-03-25T19:50:13.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 >> v1:172.21.15.71:6800/3911735624 conn(0x7fb11406df10 legacy=0x7fb1140703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:13.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 10 monclient: shutdown 2024-03-25T19:50:13.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:13.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 >> v1:172.21.15.71:6790/0 conn(0x7fb1340b1730 legacy=0x7fb13412fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:13.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 shutdown_connections 2024-03-25T19:50:13.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 >> 172.21.15.71:0/3690939008 conn(0x7fb1340617a0 msgr2=0x7fb13413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:13.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 shutdown_connections 2024-03-25T19:50:13.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 1 -- 172.21.15.71:0/3690939008 wait complete. 2024-03-25T19:50:13.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:13.195+0000 7fb13ac93640 1 librados: shutdown 2024-03-25T19:50:13.206 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:20.013 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:20.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.129+0000 7f5b97614640 10 monclient: get_monmap_and_config 2024-03-25T19:50:20.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.129+0000 7f5b97614640 10 monclient: build_initial_monmap 2024-03-25T19:50:20.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.129+0000 7f5b97614640 10 monclient: monmap: 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:50:20.130733+0000 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:50:20.130733+0000 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:50:20.130 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:20.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.130+0000 7f5b97614640 1 Processor -- start 2024-03-25T19:50:20.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.131+0000 7f5b97614640 1 -- start start 2024-03-25T19:50:20.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.131+0000 7f5b97614640 10 monclient: init 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.131+0000 7f5b97614640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.131+0000 7f5b97614640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.131+0000 7f5b97614640 10 monclient(hunting): picked mon.noname-b con 0x7f5b900b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.132+0000 7f5b97614640 10 monclient(hunting): picked mon.noname-c con 0x7f5b900b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.132+0000 7f5b97614640 10 monclient(hunting): picked mon.noname-a con 0x7f5b9014be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.132+0000 7f5b97614640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900af940 con 0x7f5b9014be90 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.132+0000 7f5b97614640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900b0a70 con 0x7f5b900b1730 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.132+0000 7f5b97614640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900b0d50 con 0x7f5b900b4d60 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.132+0000 7f5b97614640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:20.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.132+0000 7f5b97614640 10 monclient(hunting): authenticate will time out at 5426.443907s 2024-03-25T19:50:20.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.143+0000 7f5b95bb1640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5b9014be90 0x7f5b9014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:45008/0 (socket says 172.21.15.71:45008) 2024-03-25T19:50:20.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.143+0000 7f5b95bb1640 1 -- 172.21.15.71:0/3163696956 learned_addr learned my addr 172.21.15.71:0/3163696956 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055030388 0 0) 0x7f5b900af940 con 0x7f5b9014be90 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c003520 con 0x7f5b9014be90 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623019837 0 0) 0x7f5b900b0d50 con 0x7f5b900b4d60 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:20.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 10 monclient(hunting): my global_id is 14657 2024-03-25T19:50:20.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.161+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b900af940 con 0x7f5b900b4d60 2024-03-25T19:50:20.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.174+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990422366 0 0) 0x7f5b900b0a70 con 0x7f5b900b1730 2024-03-25T19:50:20.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.174+0000 7f5b87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:20.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.174+0000 7f5b87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:20.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.174+0000 7f5b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:20.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.174+0000 7f5b87fff640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:50:20.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.174+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b900b0d50 con 0x7f5b900b1730 2024-03-25T19:50:20.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3823150998 0 0) 0x7f5b900af940 con 0x7f5b900b4d60 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 >> v1:172.21.15.193:6789/0 conn(0x7f5b900b1730 legacy=0x7f5b900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 >> v1:172.21.15.71:6789/0 conn(0x7f5b9014be90 legacy=0x7f5b9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 1 monclient: found mon.noname-b 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b900b1030 con 0x7f5b900b4d60 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 10 monclient: _finish_auth 0 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5b8c001670 con 0x7f5b900b4d60 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b97614640 5 monclient: authenticate success, global_id 14657 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b97614640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:20.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b87fff640 10 monclient: dump: 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:20.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.178+0000 7f5b97614640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.192+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b8c0017d0 con 0x7f5b900b4d60 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.192+0000 7f5b87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.192+0000 7f5b87fff640 1 -- 172.21.15.71:0/3163696956 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5b8c001be0 con 0x7f5b900b4d60 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.192+0000 7f5b97614640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.192+0000 7f5b97614640 4 set_mon_vals no callback set 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.192+0000 7f5b87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.192+0000 7f5b87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.192+0000 7f5b87fff640 10 monclient: dump: 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:20.192 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:20.193 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:20.193 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:20.193 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:20.193 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:20.193 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:20.193 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:20.193 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.193+0000 7f5b97614640 10 monclient: shutdown 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.193+0000 7f5b97614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.193+0000 7f5b97614640 1 -- 172.21.15.71:0/3163696956 >> v1:172.21.15.71:6790/0 conn(0x7f5b900b4d60 legacy=0x7f5b9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.193+0000 7f5b97614640 1 -- 172.21.15.71:0/3163696956 shutdown_connections 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.193+0000 7f5b97614640 1 -- 172.21.15.71:0/3163696956 >> 172.21.15.71:0/3163696956 conn(0x7f5b900617a0 msgr2=0x7f5b90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.193+0000 7f5b97614640 1 -- 172.21.15.71:0/3163696956 shutdown_connections 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.193+0000 7f5b97614640 1 -- 172.21.15.71:0/3163696956 wait complete. 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 10 monclient: build_initial_monmap 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 10 monclient: monmap: 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:20.195 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 1 librados: starting msgr at 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 1 librados: starting objecter 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 1 Processor -- start 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 1 -- start start 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 1 librados: setting wanted keys 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 1 librados: calling monclient init 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 10 monclient: init 2024-03-25T19:50:20.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 10 monclient(hunting): picked mon.noname-c con 0x7f5b900b1730 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.194+0000 7f5b97614640 10 monclient(hunting): picked mon.noname-b con 0x7f5b900b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.195+0000 7f5b97614640 10 monclient(hunting): picked mon.noname-a con 0x7f5b9014be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.195+0000 7f5b97614640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b90141e80 con 0x7f5b9014be90 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.195+0000 7f5b97614640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b901d7570 con 0x7f5b900b4d60 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.195+0000 7f5b97614640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b901d81a0 con 0x7f5b900b1730 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.195+0000 7f5b97614640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:20.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.195+0000 7f5b97614640 10 monclient(hunting): authenticate will time out at 5426.506878s 2024-03-25T19:50:20.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.201+0000 7f5b94baf640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f5b900b4d60 0x7f5b901d7a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:43782/0 (socket says 172.21.15.71:43782) 2024-03-25T19:50:20.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.201+0000 7f5b94baf640 1 -- 172.21.15.71:0/1023466239 learned_addr learned my addr 172.21.15.71:0/1023466239 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:20.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.203+0000 7f5b85ffb640 10 client.?.objecter ms_handle_connect 0x7f5b900b4d60 2024-03-25T19:50:20.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.203+0000 7f5b85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:20.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.209+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417879470 0 0) 0x7f5b901d7570 con 0x7f5b900b4d60 2024-03-25T19:50:20.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.209+0000 7f5b85ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:20.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.209+0000 7f5b85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:20.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.209+0000 7f5b85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:20.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.209+0000 7f5b85ffb640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:50:20.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.209+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b74004eb0 con 0x7f5b900b4d60 2024-03-25T19:50:20.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.214+0000 7f5b85ffb640 10 client.?.objecter ms_handle_connect 0x7f5b900b1730 2024-03-25T19:50:20.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.214+0000 7f5b85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:20.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.215+0000 7f5b85ffb640 10 client.?.objecter ms_handle_connect 0x7f5b9014be90 2024-03-25T19:50:20.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.215+0000 7f5b85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.217+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254897780 0 0) 0x7f5b74004eb0 con 0x7f5b900b4d60 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.217+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b901d7570 con 0x7f5b900b4d60 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.217+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5b80001a10 con 0x7f5b900b4d60 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.217+0000 7f5b85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.217+0000 7f5b85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.217+0000 7f5b85ffb640 10 monclient(hunting): dump: 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:20.217 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:20.218 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:20.218 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:20.218 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:20.218 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:20.218 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:20.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.217+0000 7f5b85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:20.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1277234244 0 0) 0x7f5b901d7570 con 0x7f5b900b4d60 2024-03-25T19:50:20.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 >> v1:172.21.15.71:6790/0 conn(0x7f5b900b1730 legacy=0x7f5b90138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 >> v1:172.21.15.71:6789/0 conn(0x7f5b9014be90 legacy=0x7f5b90142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 1 monclient: found mon.b 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b901d8330 con 0x7f5b900b4d60 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 5 monclient: authenticate success, global_id 4666 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 10 monclient: _renew_subs 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b90143110 con 0x7f5b900b4d60 2024-03-25T19:50:20.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:20.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 10 monclient: _renew_subs 2024-03-25T19:50:20.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:20.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b90142480 con 0x7f5b900b4d60 2024-03-25T19:50:20.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.221+0000 7f5b97614640 1 librados: init done 2024-03-25T19:50:20.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.223+0000 7f5b97614640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:20.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.223+0000 7f5b97614640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:20.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.223+0000 7f5b97614640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:20.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.223+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b901401f0 con 0x7f5b900b4d60 2024-03-25T19:50:20.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.241+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b80001cb0 con 0x7f5b900b4d60 2024-03-25T19:50:20.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.241+0000 7f5b85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:20.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.241+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5b800038c0 con 0x7f5b900b4d60 2024-03-25T19:50:20.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.241+0000 7f5b85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:20.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.241+0000 7f5b85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:20.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.241+0000 7f5b85ffb640 10 monclient: dump: 2024-03-25T19:50:20.242 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:20.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.241+0000 7f5b877fe640 4 set_mon_vals no callback set 2024-03-25T19:50:20.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.242+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f5b80003ae0 con 0x7f5b900b4d60 2024-03-25T19:50:20.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.242+0000 7f5b953b0640 10 monclient: get_auth_request con 0x7f5b7406dd00 auth_method 0 2024-03-25T19:50:20.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.243+0000 7f5b85ffb640 10 client.4666.objecter ms_handle_connect 0x7f5b7406dd00 2024-03-25T19:50:20.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.244+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f5b800546e0 con 0x7f5b900b4d60 2024-03-25T19:50:20.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.244+0000 7f5b85ffb640 10 client.4666.objecter ms_dispatch 0x7f5b90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:20.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.244+0000 7f5b85ffb640 3 client.4666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:20.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.244+0000 7f5b85ffb640 3 client.4666.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:20.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.244+0000 7f5b85ffb640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T19:50:20.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.500+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b800501f0 con 0x7f5b900b4d60 2024-03-25T19:50:20.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.500+0000 7f5b85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:20.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.500+0000 7f5b85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:20.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.744+0000 7f5b97614640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:20.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.744+0000 7f5b97614640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:20.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.744+0000 7f5b97614640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:20.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.744+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b90141cc0 con 0x7f5b900b4d60 2024-03-25T19:50:20.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.769+0000 7f5b85ffb640 1 -- 172.21.15.71:0/1023466239 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f5b80052120 con 0x7f5b900b4d60 2024-03-25T19:50:20.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.769+0000 7f5b85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:20.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.769+0000 7f5b85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:20.769 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:50:20.769 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:20.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.771+0000 7f5b97614640 10 librados: watch_flush enter 2024-03-25T19:50:20.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 10 librados: watch_flush exit 2024-03-25T19:50:20.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T19:50:20.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 10 client.4666.objecter successfully canceled tick 2024-03-25T19:50:20.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 >> v1:172.21.15.71:6800/3911735624 conn(0x7f5b7406dd00 legacy=0x7f5b740701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:20.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 10 monclient: shutdown 2024-03-25T19:50:20.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:20.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 >> v1:172.21.15.193:6789/0 conn(0x7f5b900b4d60 legacy=0x7f5b901d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:20.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 shutdown_connections 2024-03-25T19:50:20.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 >> 172.21.15.71:0/1023466239 conn(0x7f5b900617a0 msgr2=0x7f5b900b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:20.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 shutdown_connections 2024-03-25T19:50:20.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 1 -- 172.21.15.71:0/1023466239 wait complete. 2024-03-25T19:50:20.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:20.772+0000 7f5b97614640 1 librados: shutdown 2024-03-25T19:50:20.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:27.591 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:27.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.711+0000 7f7f3de82640 10 monclient: get_monmap_and_config 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.711+0000 7f7f3de82640 10 monclient: build_initial_monmap 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.711+0000 7f7f3de82640 10 monclient: monmap: 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:50:27.712118+0000 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:50:27.712118+0000 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:50:27.711 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:27.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 1 Processor -- start 2024-03-25T19:50:27.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 1 -- start start 2024-03-25T19:50:27.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 10 monclient: init 2024-03-25T19:50:27.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 10 monclient(hunting): picked mon.noname-a con 0x7f7f3806ab80 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 10 monclient(hunting): picked mon.noname-b con 0x7f7f3806af60 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 10 monclient(hunting): picked mon.noname-c con 0x7f7f380b00b0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f380b0500 con 0x7f7f3806ab80 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3813ffa0 con 0x7f7f380b00b0 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f380af520 con 0x7f7f3806af60 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.712+0000 7f7f3de82640 10 monclient(hunting): authenticate will time out at 5434.024821s 2024-03-25T19:50:27.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.720+0000 7f7f37fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7f380b00b0 0x7f7f3814dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:42382/0 (socket says 172.21.15.71:42382) 2024-03-25T19:50:27.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.720+0000 7f7f37fff640 1 -- 172.21.15.71:0/2658279617 learned_addr learned my addr 172.21.15.71:0/2658279617 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:27.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.731+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354323628 0 0) 0x7f7f3813ffa0 con 0x7f7f380b00b0 2024-03-25T19:50:27.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.731+0000 7f7f367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:27.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.731+0000 7f7f367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:27.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.731+0000 7f7f367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:27.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.731+0000 7f7f367fc640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:50:27.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.731+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f20003520 con 0x7f7f380b00b0 2024-03-25T19:50:27.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.746+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794214640 0 0) 0x7f7f380b0500 con 0x7f7f3806ab80 2024-03-25T19:50:27.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.746+0000 7f7f367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:27.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.746+0000 7f7f367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:27.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.746+0000 7f7f367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:27.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.746+0000 7f7f367fc640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:50:27.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.746+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f3813ffa0 con 0x7f7f3806ab80 2024-03-25T19:50:27.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210336208 0 0) 0x7f7f20003520 con 0x7f7f380b00b0 2024-03-25T19:50:27.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 >> v1:172.21.15.71:6790/0 conn(0x7f7f3806af60 legacy=0x7f7f3813f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:27.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 >> v1:172.21.15.71:6789/0 conn(0x7f7f3806ab80 legacy=0x7f7f380ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:27.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 1 monclient: found mon.noname-c 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f38072890 con 0x7f7f380b00b0 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7f30001a10 con 0x7f7f380b00b0 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f367fc640 10 monclient: dump: 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:27.751 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:27.752 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:27.752 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:27.752 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:27.752 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:27.752 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:27.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f3de82640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:50:27.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.750+0000 7f7f3de82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:27.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f30001b70 con 0x7f7f380b00b0 2024-03-25T19:50:27.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:27.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f367fc640 1 -- 172.21.15.71:0/2658279617 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7f30001f80 con 0x7f7f380b00b0 2024-03-25T19:50:27.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f367fc640 10 monclient: dump: 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:27.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 4 set_mon_vals no callback set 2024-03-25T19:50:27.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 10 monclient: shutdown 2024-03-25T19:50:27.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:27.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 1 -- 172.21.15.71:0/2658279617 >> v1:172.21.15.193:6789/0 conn(0x7f7f380b00b0 legacy=0x7f7f3814dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:27.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 1 -- 172.21.15.71:0/2658279617 shutdown_connections 2024-03-25T19:50:27.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 1 -- 172.21.15.71:0/2658279617 >> 172.21.15.71:0/2658279617 conn(0x7f7f380b4a80 msgr2=0x7f7f380b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:27.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 1 -- 172.21.15.71:0/2658279617 shutdown_connections 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.755+0000 7f7f3de82640 1 -- 172.21.15.71:0/2658279617 wait complete. 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.756+0000 7f7f3de82640 10 monclient: build_initial_monmap 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.756+0000 7f7f3de82640 10 monclient: monmap: 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.756+0000 7f7f3de82640 1 librados: starting msgr at 2024-03-25T19:50:27.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.756+0000 7f7f3de82640 1 librados: starting objecter 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.756+0000 7f7f3de82640 1 Processor -- start 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 1 -- start start 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 1 librados: setting wanted keys 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 1 librados: calling monclient init 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 10 monclient: init 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 10 monclient(hunting): picked mon.noname-c con 0x7f7f3806ab80 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 10 monclient(hunting): picked mon.noname-a con 0x7f7f380b00b0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 10 monclient(hunting): picked mon.noname-b con 0x7f7f381463c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f381452d0 con 0x7f7f380b00b0 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f38145520 con 0x7f7f381463c0 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f381dacf0 con 0x7f7f3806ab80 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:27.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.757+0000 7f7f3de82640 10 monclient(hunting): authenticate will time out at 5434.069468s 2024-03-25T19:50:27.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.767+0000 7f7f377fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7f3806ab80 0x7f7f381db210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55098/0 (socket says 172.21.15.71:55098) 2024-03-25T19:50:27.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.767+0000 7f7f377fe640 1 -- 172.21.15.71:0/4148339919 learned_addr learned my addr 172.21.15.71:0/4148339919 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:27.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.768+0000 7f7f1ffff640 10 client.?.objecter ms_handle_connect 0x7f7f3806ab80 2024-03-25T19:50:27.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.768+0000 7f7f1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:27.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.774+0000 7f7f1ffff640 10 client.?.objecter ms_handle_connect 0x7f7f381463c0 2024-03-25T19:50:27.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.774+0000 7f7f1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:27.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.777+0000 7f7f1ffff640 10 client.?.objecter ms_handle_connect 0x7f7f380b00b0 2024-03-25T19:50:27.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.777+0000 7f7f1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:27.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.870+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597697392 0 0) 0x7f7f381dacf0 con 0x7f7f3806ab80 2024-03-25T19:50:27.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.870+0000 7f7f1ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:27.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.870+0000 7f7f1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:27.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.870+0000 7f7f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:27.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.870+0000 7f7f1ffff640 10 monclient(hunting): my global_id is 14666 2024-03-25T19:50:27.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.870+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f24004eb0 con 0x7f7f3806ab80 2024-03-25T19:50:27.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.882+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326288134 0 0) 0x7f7f381452d0 con 0x7f7f380b00b0 2024-03-25T19:50:27.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.882+0000 7f7f1ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:27.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.882+0000 7f7f1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:27.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.882+0000 7f7f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:27.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.882+0000 7f7f1ffff640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:50:27.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.882+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f381dacf0 con 0x7f7f380b00b0 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.890+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 148600411 0 0) 0x7f7f24004eb0 con 0x7f7f3806ab80 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.890+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f381452d0 con 0x7f7f3806ab80 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.890+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7f28001a10 con 0x7f7f3806ab80 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.890+0000 7f7f1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.890+0000 7f7f1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.890+0000 7f7f1ffff640 10 monclient(hunting): dump: 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:27.890 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:27.891 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:27.891 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:27.891 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:27.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.890+0000 7f7f1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.909+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1327132983 0 0) 0x7f7f381dacf0 con 0x7f7f380b00b0 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.909+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f24004eb0 con 0x7f7f380b00b0 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.909+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7f2c0018a0 con 0x7f7f380b00b0 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.909+0000 7f7f1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.909+0000 7f7f1ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.909+0000 7f7f1ffff640 10 monclient(hunting): dump: 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:27.909 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:27.910 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:27.910 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:27.910 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:27.910 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:27.910 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:27.910 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:27.910 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2824211358 0 0) 0x7f7f381452d0 con 0x7f7f3806ab80 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 >> v1:172.21.15.193:6789/0 conn(0x7f7f381463c0 legacy=0x7f7f381e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 >> v1:172.21.15.71:6789/0 conn(0x7f7f380b00b0 legacy=0x7f7f38145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 1 monclient: found mon.c 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f381db920 con 0x7f7f3806ab80 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 10 monclient: _finish_auth 0 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:27.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 5 monclient: authenticate success, global_id 14666 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 10 monclient: _renew_subs 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f381e3600 con 0x7f7f3806ab80 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 10 client.14666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 10 monclient: _renew_subs 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f381e3970 con 0x7f7f3806ab80 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.921+0000 7f7f3de82640 1 librados: init done 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.922+0000 7f7f3de82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.922+0000 7f7f3de82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.922+0000 7f7f3de82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:27.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:27.922+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f381467d0 con 0x7f7f3806ab80 2024-03-25T19:50:28.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.239+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f28001c50 con 0x7f7f3806ab80 2024-03-25T19:50:28.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.239+0000 7f7f1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.239+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f7f28003a60 con 0x7f7f3806ab80 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.239+0000 7f7f1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.239+0000 7f7f1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.239+0000 7f7f1ffff640 10 monclient: dump: 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:28.240 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:28.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.239+0000 7f7f35ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:28.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.240+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f7f28003c80 con 0x7f7f3806ab80 2024-03-25T19:50:28.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.241+0000 7f7f36ffd640 10 monclient: get_auth_request con 0x7f7f2406de90 auth_method 0 2024-03-25T19:50:28.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.241+0000 7f7f1ffff640 10 client.14666.objecter ms_handle_connect 0x7f7f2406de90 2024-03-25T19:50:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.246+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7f7f28054830 con 0x7f7f3806ab80 2024-03-25T19:50:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.246+0000 7f7f1ffff640 10 client.14666.objecter ms_dispatch 0x7f7f38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.246+0000 7f7f1ffff640 3 client.14666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:28.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.246+0000 7f7f1ffff640 3 client.14666.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:28.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.246+0000 7f7f1ffff640 20 client.14666.objecter dump_active .. 0 homeless 2024-03-25T19:50:28.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.246+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f28050340 con 0x7f7f3806ab80 2024-03-25T19:50:28.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.246+0000 7f7f1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:28.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.246+0000 7f7f1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:28.336 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:11.092 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.347 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:11.103 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.365 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:11.084 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.384 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:11.140 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.406 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:11.162 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.414 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:11.170 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.426 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:11.146 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.436 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:11.192 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.453 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:11.172 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:28.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.457+0000 7f7f3de82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:28.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.457+0000 7f7f3de82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:28.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.457+0000 7f7f3de82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:28.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.457+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f381dbab0 con 0x7f7f3806ab80 2024-03-25T19:50:28.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.509+0000 7f7f1ffff640 1 -- 172.21.15.71:0/4148339919 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7f7f381dbab0 con 0x7f7f3806ab80 2024-03-25T19:50:28.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.509+0000 7f7f1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:28.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.509+0000 7f7f1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:28.510 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:50:28.510 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:28.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 10 librados: watch_flush enter 2024-03-25T19:50:28.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 10 librados: watch_flush exit 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 20 client.14666.objecter shutdown clearing up homeless session... 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 10 client.14666.objecter successfully canceled tick 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 >> v1:172.21.15.71:6800/3911735624 conn(0x7f7f2406de90 legacy=0x7f7f24070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 10 monclient: shutdown 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 >> v1:172.21.15.71:6790/0 conn(0x7f7f3806ab80 legacy=0x7f7f381db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 shutdown_connections 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 >> 172.21.15.71:0/4148339919 conn(0x7f7f380b4a80 msgr2=0x7f7f38070310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 shutdown_connections 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 1 -- 172.21.15.71:0/4148339919 wait complete. 2024-03-25T19:50:28.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:28.512+0000 7f7f3de82640 1 librados: shutdown 2024-03-25T19:50:28.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:35.329 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:35.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.449+0000 7fc339296640 10 monclient: get_monmap_and_config 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.449+0000 7fc339296640 10 monclient: build_initial_monmap 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.449+0000 7fc339296640 10 monclient: monmap: 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:50:35.451100+0000 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:50:35.451100+0000 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:50:35.450 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.450+0000 7fc339296640 1 Processor -- start 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.450+0000 7fc339296640 1 -- start start 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.450+0000 7fc339296640 10 monclient: init 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.450+0000 7fc339296640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.450+0000 7fc339296640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.451+0000 7fc339296640 10 monclient(hunting): picked mon.noname-a con 0x7fc3340a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.451+0000 7fc339296640 10 monclient(hunting): picked mon.noname-b con 0x7fc33413f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.451+0000 7fc339296640 10 monclient(hunting): picked mon.noname-c con 0x7fc3341430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.451+0000 7fc339296640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc334142520 con 0x7fc3340a98f0 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.451+0000 7fc339296640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3340b28a0 con 0x7fc3341430c0 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.451+0000 7fc339296640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3340b2a30 con 0x7fc33413f710 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.451+0000 7fc339296640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:35.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.451+0000 7fc339296640 10 monclient(hunting): authenticate will time out at 5441.764002s 2024-03-25T19:50:35.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.459+0000 7fc332ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc3340a98f0 0x7fc3340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44420/0 (socket says 172.21.15.71:44420) 2024-03-25T19:50:35.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.459+0000 7fc332ffd640 1 -- 172.21.15.71:0/1296583284 learned_addr learned my addr 172.21.15.71:0/1296583284 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:35.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.472+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142726659 0 0) 0x7fc334142520 con 0x7fc3340a98f0 2024-03-25T19:50:35.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.472+0000 7fc3327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:35.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.472+0000 7fc3327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:35.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.472+0000 7fc3327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:35.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.472+0000 7fc3327fc640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:50:35.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.472+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc320003520 con 0x7fc3340a98f0 2024-03-25T19:50:35.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.475+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710075002 0 0) 0x7fc3340b28a0 con 0x7fc3341430c0 2024-03-25T19:50:35.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.475+0000 7fc3327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:35.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.475+0000 7fc3327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:35.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.475+0000 7fc3327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:35.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.475+0000 7fc3327fc640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:50:35.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.475+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc334142520 con 0x7fc3341430c0 2024-03-25T19:50:35.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873219 0 0) 0x7fc320003520 con 0x7fc3340a98f0 2024-03-25T19:50:35.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 >> v1:172.21.15.71:6790/0 conn(0x7fc33413f710 legacy=0x7fc3341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 >> v1:172.21.15.193:6789/0 conn(0x7fc3341430c0 legacy=0x7fc33414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 1 monclient: found mon.noname-a 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340b44c0 con 0x7fc3340a98f0 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc324001690 con 0x7fc3340a98f0 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc339296640 5 monclient: authenticate success, global_id 4677 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc339296640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:35.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc3327fc640 10 monclient: dump: 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:35.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.481+0000 7fc339296640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:35.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.489+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3240018b0 con 0x7fc3340a98f0 2024-03-25T19:50:35.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.489+0000 7fc3327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:35.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.489+0000 7fc3327fc640 1 -- 172.21.15.71:0/1296583284 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc324001c40 con 0x7fc3340a98f0 2024-03-25T19:50:35.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.489+0000 7fc3327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:35.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.489+0000 7fc3327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:35.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.489+0000 7fc3327fc640 10 monclient: dump: 2024-03-25T19:50:35.490 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.489+0000 7fc339296640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:35.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.489+0000 7fc339296640 4 set_mon_vals no callback set 2024-03-25T19:50:35.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.490+0000 7fc339296640 10 monclient: shutdown 2024-03-25T19:50:35.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.490+0000 7fc339296640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:35.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.490+0000 7fc339296640 1 -- 172.21.15.71:0/1296583284 >> v1:172.21.15.71:6789/0 conn(0x7fc3340a98f0 legacy=0x7fc3340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:35.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.490+0000 7fc339296640 1 -- 172.21.15.71:0/1296583284 shutdown_connections 2024-03-25T19:50:35.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.490+0000 7fc339296640 1 -- 172.21.15.71:0/1296583284 >> 172.21.15.71:0/1296583284 conn(0x7fc3340b7ea0 msgr2=0x7fc3340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:35.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.490+0000 7fc339296640 1 -- 172.21.15.71:0/1296583284 shutdown_connections 2024-03-25T19:50:35.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.490+0000 7fc339296640 1 -- 172.21.15.71:0/1296583284 wait complete. 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.491+0000 7fc339296640 10 monclient: build_initial_monmap 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.491+0000 7fc339296640 10 monclient: monmap: 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:50:35.494 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.491+0000 7fc339296640 1 librados: starting msgr at 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.491+0000 7fc339296640 1 librados: starting objecter 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.491+0000 7fc339296640 1 Processor -- start 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 1 -- start start 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 1 librados: setting wanted keys 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 1 librados: calling monclient init 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 10 monclient: init 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 10 monclient(hunting): picked mon.noname-b con 0x7fc3340a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 10 monclient(hunting): picked mon.noname-c con 0x7fc33413f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 10 monclient(hunting): picked mon.noname-a con 0x7fc3341430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3341369b0 con 0x7fc3341430c0 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3341366b0 con 0x7fc3340a98f0 2024-03-25T19:50:35.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc334133ad0 con 0x7fc33413f710 2024-03-25T19:50:35.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:35.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.492+0000 7fc339296640 10 monclient(hunting): authenticate will time out at 5441.805580s 2024-03-25T19:50:35.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.496+0000 7fc332ffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc3340a98f0 0x7fc3341371e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:47152/0 (socket says 172.21.15.71:47152) 2024-03-25T19:50:35.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.496+0000 7fc332ffd640 1 -- 172.21.15.71:0/1654157296 learned_addr learned my addr 172.21.15.71:0/1654157296 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:35.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.498+0000 7fc32b7fe640 10 client.?.objecter ms_handle_connect 0x7fc3340a98f0 2024-03-25T19:50:35.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.498+0000 7fc32b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:35.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.506+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561815583 0 0) 0x7fc3341366b0 con 0x7fc3340a98f0 2024-03-25T19:50:35.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.506+0000 7fc32b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:35.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.506+0000 7fc32b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:35.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.506+0000 7fc32b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:35.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.506+0000 7fc32b7fe640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:50:35.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.506+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc314004eb0 con 0x7fc3340a98f0 2024-03-25T19:50:35.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.511+0000 7fc32b7fe640 10 client.?.objecter ms_handle_connect 0x7fc33413f710 2024-03-25T19:50:35.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.511+0000 7fc32b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.518+0000 7fc32b7fe640 10 client.?.objecter ms_handle_connect 0x7fc3341430c0 2024-03-25T19:50:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.518+0000 7fc32b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.519+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2516874584 0 0) 0x7fc314004eb0 con 0x7fc3340a98f0 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.519+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3341366b0 con 0x7fc3340a98f0 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.519+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc3240016d0 con 0x7fc3340a98f0 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.519+0000 7fc32b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.519+0000 7fc32b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.519+0000 7fc32b7fe640 10 monclient(hunting): dump: 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:35.520 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:35.521 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:35.521 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:35.521 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:35.521 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:35.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.519+0000 7fc32b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.525+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137812953 0 0) 0x7fc334133ad0 con 0x7fc33413f710 2024-03-25T19:50:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.525+0000 7fc32b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.525+0000 7fc32b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.525+0000 7fc32b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.525+0000 7fc32b7fe640 10 monclient(hunting): my global_id is 14672 2024-03-25T19:50:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.525+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc314004eb0 con 0x7fc33413f710 2024-03-25T19:50:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2805888597 0 0) 0x7fc3341366b0 con 0x7fc3340a98f0 2024-03-25T19:50:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 >> v1:172.21.15.71:6790/0 conn(0x7fc33413f710 legacy=0x7fc334132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 >> v1:172.21.15.71:6789/0 conn(0x7fc3341430c0 legacy=0x7fc3341333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 1 monclient: found mon.b 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334136e40 con 0x7fc3340a98f0 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc32b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc339296640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc339296640 10 monclient: _renew_subs 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.534+0000 7fc339296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.535+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3341378f0 con 0x7fc3340a98f0 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.535+0000 7fc339296640 10 client.4678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.535+0000 7fc339296640 10 monclient: _renew_subs 2024-03-25T19:50:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.535+0000 7fc339296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.535+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3340b1480 con 0x7fc3340a98f0 2024-03-25T19:50:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.535+0000 7fc339296640 1 librados: init done 2024-03-25T19:50:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.535+0000 7fc339296640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.536+0000 7fc339296640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.536+0000 7fc339296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.536+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc334137030 con 0x7fc3340a98f0 2024-03-25T19:50:35.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.810+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc324004a90 con 0x7fc3340a98f0 2024-03-25T19:50:35.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.811+0000 7fc32b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.811+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc324004e60 con 0x7fc3340a98f0 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.811+0000 7fc32b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.811+0000 7fc32b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.811+0000 7fc32b7fe640 10 monclient: dump: 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:35.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.811+0000 7fc331ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:35.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.812+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fc324005080 con 0x7fc3340a98f0 2024-03-25T19:50:35.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.812+0000 7fc32bfff640 10 monclient: get_auth_request con 0x7fc31406de20 auth_method 0 2024-03-25T19:50:35.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.812+0000 7fc32b7fe640 10 client.4678.objecter ms_handle_connect 0x7fc31406de20 2024-03-25T19:50:35.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.820+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fc324054990 con 0x7fc3340a98f0 2024-03-25T19:50:35.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.820+0000 7fc32b7fe640 10 client.4678.objecter ms_dispatch 0x7fc334000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:35.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.820+0000 7fc32b7fe640 3 client.4678.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:35.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.820+0000 7fc32b7fe640 3 client.4678.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:35.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.820+0000 7fc32b7fe640 20 client.4678.objecter dump_active .. 0 homeless 2024-03-25T19:50:35.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.820+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3240504a0 con 0x7fc3340a98f0 2024-03-25T19:50:35.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.820+0000 7fc32b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:35.821 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:35.820+0000 7fc32b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:36.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.042+0000 7fc339296640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:36.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.042+0000 7fc339296640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:36.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.042+0000 7fc339296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:36.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.042+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3340627b0 con 0x7fc3340a98f0 2024-03-25T19:50:36.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.067+0000 7fc32b7fe640 1 -- 172.21.15.71:0/1654157296 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1058172006) 0x7fc3240523d0 con 0x7fc3340a98f0 2024-03-25T19:50:36.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.067+0000 7fc32b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:36.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.067+0000 7fc32b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:36.069 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:50:36.069 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:40:44.629895+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:44.629895+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.069+0000 7fc339296640 10 librados: watch_flush enter 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 10 librados: watch_flush exit 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 20 client.4678.objecter shutdown clearing up homeless session... 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 10 client.4678.objecter successfully canceled tick 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 >> v1:172.21.15.71:6800/3911735624 conn(0x7fc31406de20 legacy=0x7fc3140702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 10 monclient: shutdown 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 >> v1:172.21.15.193:6789/0 conn(0x7fc3340a98f0 legacy=0x7fc3341371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 shutdown_connections 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 >> 172.21.15.71:0/1654157296 conn(0x7fc3340b7ea0 msgr2=0x7fc334074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:36.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 shutdown_connections 2024-03-25T19:50:36.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 1 -- 172.21.15.71:0/1654157296 wait complete. 2024-03-25T19:50:36.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:36.070+0000 7fc339296640 1 librados: shutdown 2024-03-25T19:50:36.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:42.890 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.004+0000 7fce7f84d640 10 monclient: get_monmap_and_config 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.004+0000 7fce7f84d640 10 monclient: build_initial_monmap 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.004+0000 7fce7f84d640 10 monclient: monmap: 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:50:43.005364+0000 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:50:43.005364+0000 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:43.004 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:50:43.005 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:50:43.005 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.005+0000 7fce7f84d640 1 Processor -- start 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.005+0000 7fce7f84d640 1 -- start start 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.005+0000 7fce7f84d640 10 monclient: init 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 10 monclient(hunting): picked mon.noname-b con 0x7fce780a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 10 monclient(hunting): picked mon.noname-a con 0x7fce7813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 10 monclient(hunting): picked mon.noname-c con 0x7fce781430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce78142520 con 0x7fce7813f710 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce780b28a0 con 0x7fce781430c0 2024-03-25T19:50:43.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce780b2a30 con 0x7fce780a98f0 2024-03-25T19:50:43.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:43.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.006+0000 7fce7f84d640 10 monclient(hunting): authenticate will time out at 5449.318138s 2024-03-25T19:50:43.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.015+0000 7fce7ddea640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fce781430c0 0x7fce7814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:35932/0 (socket says 172.21.15.71:35932) 2024-03-25T19:50:43.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.015+0000 7fce7ddea640 1 -- 172.21.15.71:0/1641753160 learned_addr learned my addr 172.21.15.71:0/1641753160 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:43.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.028+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589768516 0 0) 0x7fce780b28a0 con 0x7fce781430c0 2024-03-25T19:50:43.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.028+0000 7fce6ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:43.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.028+0000 7fce6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:43.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.028+0000 7fce6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:43.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.028+0000 7fce6ffff640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:50:43.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.028+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce64003520 con 0x7fce781430c0 2024-03-25T19:50:43.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.033+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466950618 0 0) 0x7fce78142520 con 0x7fce7813f710 2024-03-25T19:50:43.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.033+0000 7fce6ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:43.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.033+0000 7fce6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:43.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.033+0000 7fce6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:43.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.033+0000 7fce6ffff640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:50:43.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.033+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce780b28a0 con 0x7fce7813f710 2024-03-25T19:50:43.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2008509307 0 0) 0x7fce64003520 con 0x7fce781430c0 2024-03-25T19:50:43.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 >> v1:172.21.15.71:6790/0 conn(0x7fce780a98f0 legacy=0x7fce780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:50:43.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 >> v1:172.21.15.71:6789/0 conn(0x7fce7813f710 legacy=0x7fce781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:43.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:43.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 1 monclient: found mon.noname-c 2024-03-25T19:50:43.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:43.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:50:43.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce780b44c0 con 0x7fce781430c0 2024-03-25T19:50:43.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 10 monclient: _finish_auth 0 2024-03-25T19:50:43.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:43.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:43.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fce70001a10 con 0x7fce781430c0 2024-03-25T19:50:43.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:43.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:43.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce6ffff640 10 monclient: dump: 2024-03-25T19:50:43.041 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:43.041 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:43.041 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce7f84d640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:50:43.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.037+0000 7fce7f84d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:43.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce70001b70 con 0x7fce781430c0 2024-03-25T19:50:43.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:43.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce6ffff640 1 -- 172.21.15.71:0/1641753160 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fce70001f80 con 0x7fce781430c0 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce6ffff640 10 monclient: dump: 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce7f84d640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:43.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce7f84d640 4 set_mon_vals no callback set 2024-03-25T19:50:43.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce7f84d640 10 monclient: shutdown 2024-03-25T19:50:43.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce7f84d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.055+0000 7fce7f84d640 1 -- 172.21.15.71:0/1641753160 >> v1:172.21.15.193:6789/0 conn(0x7fce781430c0 legacy=0x7fce7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.056+0000 7fce7f84d640 1 -- 172.21.15.71:0/1641753160 shutdown_connections 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.056+0000 7fce7f84d640 1 -- 172.21.15.71:0/1641753160 >> 172.21.15.71:0/1641753160 conn(0x7fce780b7ea0 msgr2=0x7fce780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.056+0000 7fce7f84d640 1 -- 172.21.15.71:0/1641753160 shutdown_connections 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.056+0000 7fce7f84d640 1 -- 172.21.15.71:0/1641753160 wait complete. 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.056+0000 7fce7f84d640 10 monclient: build_initial_monmap 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.056+0000 7fce7f84d640 10 monclient: monmap: 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:43.058 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 librados: starting msgr at 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 librados: starting objecter 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 Processor -- start 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 -- start start 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 librados: setting wanted keys 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 librados: calling monclient init 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 10 monclient: init 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 10 monclient(hunting): picked mon.noname-c con 0x7fce780a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 10 monclient(hunting): picked mon.noname-a con 0x7fce7813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 10 monclient(hunting): picked mon.noname-b con 0x7fce781430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce780ab8b0 con 0x7fce7813f710 2024-03-25T19:50:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce781dc070 con 0x7fce781430c0 2024-03-25T19:50:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce78138bf0 con 0x7fce780a98f0 2024-03-25T19:50:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.057+0000 7fce7f84d640 10 monclient(hunting): authenticate will time out at 5449.369686s 2024-03-25T19:50:43.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.068+0000 7fce7cde8640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fce7813f710 0x7fce781db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60236/0 (socket says 172.21.15.71:60236) 2024-03-25T19:50:43.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.068+0000 7fce7cde8640 1 -- 172.21.15.71:0/793056301 learned_addr learned my addr 172.21.15.71:0/793056301 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.075+0000 7fce6dffb640 10 client.?.objecter ms_handle_connect 0x7fce7813f710 2024-03-25T19:50:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.075+0000 7fce6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.075+0000 7fce6dffb640 10 client.?.objecter ms_handle_connect 0x7fce780a98f0 2024-03-25T19:50:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.075+0000 7fce6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.080+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186962521 0 0) 0x7fce78138bf0 con 0x7fce780a98f0 2024-03-25T19:50:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 10 monclient(hunting): my global_id is 14675 2024-03-25T19:50:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce5c004eb0 con 0x7fce780a98f0 2024-03-25T19:50:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350510573 0 0) 0x7fce780ab8b0 con 0x7fce7813f710 2024-03-25T19:50:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:50:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.081+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce78138bf0 con 0x7fce7813f710 2024-03-25T19:50:43.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.086+0000 7fce6dffb640 10 client.?.objecter ms_handle_connect 0x7fce781430c0 2024-03-25T19:50:43.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.086+0000 7fce6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:43.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.099+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3355232570 0 0) 0x7fce5c004eb0 con 0x7fce780a98f0 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.099+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce780ab8b0 con 0x7fce780a98f0 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.099+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fce74001870 con 0x7fce780a98f0 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.099+0000 7fce6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.099+0000 7fce6dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.099+0000 7fce6dffb640 10 monclient(hunting): dump: 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:43.099 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:43.100 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:43.100 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:43.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.099+0000 7fce6dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:43.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.103+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199736664 0 0) 0x7fce781dc070 con 0x7fce781430c0 2024-03-25T19:50:43.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.103+0000 7fce6dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:43.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.103+0000 7fce6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:43.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.103+0000 7fce6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:43.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.103+0000 7fce6dffb640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:50:43.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.103+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce5c004eb0 con 0x7fce781430c0 2024-03-25T19:50:43.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3348368821 0 0) 0x7fce780ab8b0 con 0x7fce780a98f0 2024-03-25T19:50:43.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 >> v1:172.21.15.193:6789/0 conn(0x7fce781430c0 legacy=0x7fce781db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:43.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 >> v1:172.21.15.71:6789/0 conn(0x7fce7813f710 legacy=0x7fce781db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:43.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:43.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 1 monclient: found mon.c 2024-03-25T19:50:43.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:43.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:43.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce78138ed0 con 0x7fce780a98f0 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 5 monclient: authenticate success, global_id 14675 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 10 monclient: _renew_subs 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce781391b0 con 0x7fce780a98f0 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 10 client.14675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 10 monclient: _renew_subs 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce78136d30 con 0x7fce780a98f0 2024-03-25T19:50:43.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.124+0000 7fce7f84d640 1 librados: init done 2024-03-25T19:50:43.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.125+0000 7fce7f84d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:43.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.125+0000 7fce7f84d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:43.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.125+0000 7fce7f84d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:43.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.125+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce780b3ef0 con 0x7fce780a98f0 2024-03-25T19:50:43.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.422+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce740035f0 con 0x7fce780a98f0 2024-03-25T19:50:43.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.422+0000 7fce6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:43.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.422+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fce74001d80 con 0x7fce780a98f0 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.422+0000 7fce6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.422+0000 7fce6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.422+0000 7fce6dffb640 10 monclient: dump: 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.422+0000 7fce6f7fe640 4 set_mon_vals no callback set 2024-03-25T19:50:43.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.423+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fce74019830 con 0x7fce780a98f0 2024-03-25T19:50:43.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.423+0000 7fce7cde8640 10 monclient: get_auth_request con 0x7fce5c06de00 auth_method 0 2024-03-25T19:50:43.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.423+0000 7fce6dffb640 10 client.14675.objecter ms_handle_connect 0x7fce5c06de00 2024-03-25T19:50:43.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.431+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3974953490 0 0) 0x7fce74053560 con 0x7fce780a98f0 2024-03-25T19:50:43.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.431+0000 7fce6dffb640 10 client.14675.objecter ms_dispatch 0x7fce78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:43.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.431+0000 7fce6dffb640 3 client.14675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:43.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.431+0000 7fce6dffb640 3 client.14675.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:43.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.431+0000 7fce6dffb640 20 client.14675.objecter dump_active .. 0 homeless 2024-03-25T19:50:43.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.431+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce7404eea0 con 0x7fce780a98f0 2024-03-25T19:50:43.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.431+0000 7fce6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:43.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.431+0000 7fce6dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:43.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.656+0000 7fce7f84d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:43.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.656+0000 7fce7f84d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:43.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.656+0000 7fce7f84d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:50:43.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.656+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce780627b0 con 0x7fce780a98f0 2024-03-25T19:50:43.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.744+0000 7fce6dffb640 1 -- 172.21.15.71:0/793056301 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7fce74050fa0 con 0x7fce780a98f0 2024-03-25T19:50:43.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.744+0000 7fce6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:43.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.744+0000 7fce6dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:43.744 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:50:43.745 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.746+0000 7fce7f84d640 10 librados: watch_flush enter 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.746+0000 7fce7f84d640 10 librados: watch_flush exit 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.746+0000 7fce7f84d640 20 client.14675.objecter shutdown clearing up homeless session... 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.746+0000 7fce7f84d640 10 client.14675.objecter successfully canceled tick 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 >> v1:172.21.15.71:6800/3911735624 conn(0x7fce5c06de00 legacy=0x7fce5c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 10 monclient: shutdown 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 >> v1:172.21.15.71:6790/0 conn(0x7fce780a98f0 legacy=0x7fce781375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 shutdown_connections 2024-03-25T19:50:43.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 >> 172.21.15.71:0/793056301 conn(0x7fce780b7ea0 msgr2=0x7fce78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:43.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 shutdown_connections 2024-03-25T19:50:43.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 1 -- 172.21.15.71:0/793056301 wait complete. 2024-03-25T19:50:43.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:43.747+0000 7fce7f84d640 1 librados: shutdown 2024-03-25T19:50:43.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:48.215 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:30.971 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.220 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:30.977 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.252 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:31.009 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.272 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:31.028 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.290 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:31.009 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.294 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:31.050 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.300 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:31.019 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.330 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:31.086 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.330 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:31.049 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.564 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.683+0000 7f648794d640 10 monclient: get_monmap_and_config 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.683+0000 7f648794d640 10 monclient: build_initial_monmap 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.683+0000 7f648794d640 10 monclient: monmap: 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:50:50.684438+0000 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:50:50.684438+0000 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:50.683 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:50.684 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:50.684 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:50:50.684 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:50:50.684 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:50.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 1 Processor -- start 2024-03-25T19:50:50.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 1 -- start start 2024-03-25T19:50:50.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 10 monclient: init 2024-03-25T19:50:50.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:50.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:50.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 10 monclient(hunting): picked mon.noname-c con 0x7f64800ae6e0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 10 monclient(hunting): picked mon.noname-b con 0x7f648006f080 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 10 monclient(hunting): picked mon.noname-a con 0x7f64800704e0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64800718d0 con 0x7f64800704e0 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648006fa30 con 0x7f64800ae6e0 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6480070090 con 0x7f648006f080 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.684+0000 7f648794d640 10 monclient(hunting): authenticate will time out at 5456.996812s 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.685+0000 7f6484ee8640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f648006f080 0x7f6480071c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57634/0 (socket says 172.21.15.71:57634) 2024-03-25T19:50:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.685+0000 7f6484ee8640 1 -- 172.21.15.71:0/950101723 learned_addr learned my addr 172.21.15.71:0/950101723 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.730+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932818779 0 0) 0x7f64800718d0 con 0x7f64800704e0 2024-03-25T19:50:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.730+0000 7f6477fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.730+0000 7f6477fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.730+0000 7f6477fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.730+0000 7f6477fff640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:50:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.730+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f646c003520 con 0x7f64800704e0 2024-03-25T19:50:50.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4092240666 0 0) 0x7f646c003520 con 0x7f64800704e0 2024-03-25T19:50:50.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 >> v1:172.21.15.71:6790/0 conn(0x7f648006f080 legacy=0x7f6480071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:50.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 >> v1:172.21.15.193:6789/0 conn(0x7f64800ae6e0 legacy=0x7f648013c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 1 monclient: found mon.noname-a 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648006f510 con 0x7f64800704e0 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 10 monclient: _finish_auth 0 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6478001a10 con 0x7f64800704e0 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f6477fff640 10 monclient: dump: 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:50.745 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f648794d640 5 monclient: authenticate success, global_id 4689 2024-03-25T19:50:50.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.744+0000 7f648794d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:50.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6478001b70 con 0x7f64800704e0 2024-03-25T19:50:50.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f6477fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f6477fff640 1 -- 172.21.15.71:0/950101723 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6478001f80 con 0x7f64800704e0 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f648794d640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f648794d640 4 set_mon_vals no callback set 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f6477fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f6477fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f6477fff640 10 monclient: dump: 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:50.771 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:50.772 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:50.772 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:50.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f648794d640 10 monclient: shutdown 2024-03-25T19:50:50.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f648794d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:50.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f648794d640 1 -- 172.21.15.71:0/950101723 >> v1:172.21.15.71:6789/0 conn(0x7f64800704e0 legacy=0x7f648014a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:50.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f648794d640 1 -- 172.21.15.71:0/950101723 shutdown_connections 2024-03-25T19:50:50.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f648794d640 1 -- 172.21.15.71:0/950101723 >> 172.21.15.71:0/950101723 conn(0x7f648006b630 msgr2=0x7f64800b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:50.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.771+0000 7f648794d640 1 -- 172.21.15.71:0/950101723 shutdown_connections 2024-03-25T19:50:50.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.772+0000 7f648794d640 1 -- 172.21.15.71:0/950101723 wait complete. 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.772+0000 7f648794d640 10 monclient: build_initial_monmap 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.772+0000 7f648794d640 10 monclient: monmap: 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:50.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.773+0000 7f648794d640 1 librados: starting msgr at 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.773+0000 7f648794d640 1 librados: starting objecter 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.773+0000 7f648794d640 1 Processor -- start 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.773+0000 7f648794d640 1 -- start start 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.773+0000 7f648794d640 1 librados: setting wanted keys 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.773+0000 7f648794d640 1 librados: calling monclient init 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.773+0000 7f648794d640 10 monclient: init 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 10 monclient(hunting): picked mon.noname-a con 0x7f64800704e0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 10 monclient(hunting): picked mon.noname-c con 0x7f64800ae6e0 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 10 monclient(hunting): picked mon.noname-b con 0x7f64800aef70 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64800aa270 con 0x7f64800704e0 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64800aa550 con 0x7f64800aef70 2024-03-25T19:50:50.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64801367e0 con 0x7f64800ae6e0 2024-03-25T19:50:50.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:50.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.774+0000 7f648794d640 10 monclient(hunting): authenticate will time out at 5457.086193s 2024-03-25T19:50:50.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.780+0000 7f64856e9640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f64800704e0 0x7f6480142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51946/0 (socket says 172.21.15.71:51946) 2024-03-25T19:50:50.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.780+0000 7f64856e9640 1 -- 172.21.15.71:0/1653274421 learned_addr learned my addr 172.21.15.71:0/1653274421 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:50.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.781+0000 7f6475ffb640 10 client.?.objecter ms_handle_connect 0x7f64800704e0 2024-03-25T19:50:50.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.781+0000 7f6475ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:50.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.790+0000 7f6475ffb640 10 client.?.objecter ms_handle_connect 0x7f64800ae6e0 2024-03-25T19:50:50.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.790+0000 7f6475ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:50.795 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.795+0000 7f6475ffb640 10 client.?.objecter ms_handle_connect 0x7f64800aef70 2024-03-25T19:50:50.795 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.795+0000 7f6475ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:50.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.798+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053912833 0 0) 0x7f64800aa270 con 0x7f64800704e0 2024-03-25T19:50:50.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.798+0000 7f6475ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:50.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.798+0000 7f6475ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:50.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.798+0000 7f6475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:50.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.798+0000 7f6475ffb640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:50:50.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.798+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6464004eb0 con 0x7f64800704e0 2024-03-25T19:50:50.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.800+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227081989 0 0) 0x7f64800aa550 con 0x7f64800aef70 2024-03-25T19:50:50.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.800+0000 7f6475ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:50.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.800+0000 7f6475ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:50.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.800+0000 7f6475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:50.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.800+0000 7f6475ffb640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:50:50.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.800+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64800aa270 con 0x7f64800aef70 2024-03-25T19:50:50.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.801+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713663472 0 0) 0x7f64801367e0 con 0x7f64800ae6e0 2024-03-25T19:50:50.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.801+0000 7f6475ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:50.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.801+0000 7f6475ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:50.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.801+0000 7f6475ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:50.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.801+0000 7f6475ffb640 10 monclient(hunting): my global_id is 14681 2024-03-25T19:50:50.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.801+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64800aa550 con 0x7f64800ae6e0 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.809+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2164226125 0 0) 0x7f64800aa270 con 0x7f64800aef70 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.809+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64801367e0 con 0x7f64800aef70 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.809+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6478001ec0 con 0x7f64800aef70 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.809+0000 7f6475ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.809+0000 7f6475ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.809+0000 7f6475ffb640 10 monclient(hunting): dump: 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:50.809 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:50.810 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:50.810 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:50.810 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:50.810 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:50.810 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:50.810 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:50.810 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:50.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.809+0000 7f6475ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:50.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2808454412 0 0) 0x7f64801367e0 con 0x7f64800aef70 2024-03-25T19:50:50.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 >> v1:172.21.15.71:6790/0 conn(0x7f64800ae6e0 legacy=0x7f6480142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 >> v1:172.21.15.71:6789/0 conn(0x7f64800704e0 legacy=0x7f6480142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 1 monclient: found mon.b 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64800ad0a0 con 0x7f64800aef70 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f6475ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:50:50.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 10 monclient: _renew_subs 2024-03-25T19:50:50.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:50.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64801e3a10 con 0x7f64800aef70 2024-03-25T19:50:50.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:50.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 10 monclient: _renew_subs 2024-03-25T19:50:50.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:50.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64801e3d80 con 0x7f64800aef70 2024-03-25T19:50:50.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.814+0000 7f648794d640 1 librados: init done 2024-03-25T19:50:50.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.815+0000 7f648794d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:50.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.815+0000 7f648794d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:50.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.815+0000 7f648794d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:50.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:50.815+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64800627b0 con 0x7f64800aef70 2024-03-25T19:50:51.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.084+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6478003c20 con 0x7f64800aef70 2024-03-25T19:50:51.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.084+0000 7f6475ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:51.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.084+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6478003f30 con 0x7f64800aef70 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.084+0000 7f6475ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.084+0000 7f6475ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.084+0000 7f6475ffb640 10 monclient: dump: 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:51.085 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:51.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.084+0000 7f64777fe640 4 set_mon_vals no callback set 2024-03-25T19:50:51.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.085+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f6478019f00 con 0x7f64800aef70 2024-03-25T19:50:51.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.085+0000 7f64856e9640 10 monclient: get_auth_request con 0x7f646406ddd0 auth_method 0 2024-03-25T19:50:51.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.086+0000 7f6475ffb640 10 client.4690.objecter ms_handle_connect 0x7f646406ddd0 2024-03-25T19:50:51.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.089+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7f6478053a20 con 0x7f64800aef70 2024-03-25T19:50:51.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.089+0000 7f6475ffb640 10 client.4690.objecter ms_dispatch 0x7f6480000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:50:51.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.089+0000 7f6475ffb640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:50:51.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.089+0000 7f6475ffb640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:50:51.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.090+0000 7f6475ffb640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T19:50:51.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.090+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f647804f530 con 0x7f64800aef70 2024-03-25T19:50:51.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.090+0000 7f6475ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:51.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.090+0000 7f6475ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:51.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.351+0000 7f648794d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:51.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.351+0000 7f648794d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:51.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.351+0000 7f648794d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:51.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.351+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6480074d80 con 0x7f64800aef70 2024-03-25T19:50:51.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.374+0000 7f6475ffb640 1 -- 172.21.15.71:0/1653274421 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7f6480074d80 con 0x7f64800aef70 2024-03-25T19:50:51.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.374+0000 7f6475ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:51.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.374+0000 7f6475ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:51.374 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:50:51.374 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:51.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.376+0000 7f648794d640 10 librados: watch_flush enter 2024-03-25T19:50:51.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.376+0000 7f648794d640 10 librados: watch_flush exit 2024-03-25T19:50:51.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.376+0000 7f648794d640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T19:50:51.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.376+0000 7f648794d640 10 client.4690.objecter successfully canceled tick 2024-03-25T19:50:51.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 >> v1:172.21.15.71:6800/3911735624 conn(0x7f646406ddd0 legacy=0x7f6464070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:51.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 10 monclient: shutdown 2024-03-25T19:50:51.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:51.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 >> v1:172.21.15.193:6789/0 conn(0x7f64800aef70 legacy=0x7f64801e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:51.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 shutdown_connections 2024-03-25T19:50:51.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 >> 172.21.15.71:0/1653274421 conn(0x7f648006b630 msgr2=0x7f648013c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:51.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 shutdown_connections 2024-03-25T19:50:51.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 1 -- 172.21.15.71:0/1653274421 wait complete. 2024-03-25T19:50:51.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:51.377+0000 7f648794d640 1 librados: shutdown 2024-03-25T19:50:51.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:58.196 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:58.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.308+0000 7f5441ed6640 10 monclient: get_monmap_and_config 2024-03-25T19:50:58.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.308+0000 7f5441ed6640 10 monclient: build_initial_monmap 2024-03-25T19:50:58.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.308+0000 7f5441ed6640 10 monclient: monmap: 2024-03-25T19:50:58.308 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:58.308 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:58.309 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:50:58.309617+0000 2024-03-25T19:50:58.309 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:50:58.309617+0000 2024-03-25T19:50:58.309 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:58.309 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:58.309 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:58.309 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:50:58.309 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:50:58.309 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:58.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.309+0000 7f5441ed6640 1 Processor -- start 2024-03-25T19:50:58.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.309+0000 7f5441ed6640 1 -- start start 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.309+0000 7f5441ed6640 10 monclient: init 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 10 monclient(hunting): picked mon.noname-b con 0x7f543c0a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 10 monclient(hunting): picked mon.noname-c con 0x7f543c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 10 monclient(hunting): picked mon.noname-a con 0x7f543c1430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c142520 con 0x7f543c1430c0 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c0b28a0 con 0x7f543c13f710 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c0b2a30 con 0x7f543c0a98f0 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:58.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.310+0000 7f5441ed6640 10 monclient(hunting): authenticate will time out at 5464.622441s 2024-03-25T19:50:58.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.320+0000 7f543b7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f543c0a98f0 0x7f543c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39306/0 (socket says 172.21.15.71:39306) 2024-03-25T19:50:58.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.320+0000 7f543b7fe640 1 -- 172.21.15.71:0/2258210079 learned_addr learned my addr 172.21.15.71:0/2258210079 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102872895 0 0) 0x7f543c0b28a0 con 0x7f543c13f710 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5424003520 con 0x7f543c13f710 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293145595 0 0) 0x7f543c0b2a30 con 0x7f543c0a98f0 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:58.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:58.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:58.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 10 monclient(hunting): my global_id is 14684 2024-03-25T19:50:58.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.331+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c0b28a0 con 0x7f543c0a98f0 2024-03-25T19:50:58.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.342+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313102345 0 0) 0x7f543c142520 con 0x7f543c1430c0 2024-03-25T19:50:58.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.342+0000 7f543a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:58.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.342+0000 7f543a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:58.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.342+0000 7f543a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:58.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.342+0000 7f543a7fc640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:50:58.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.342+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c0b2a30 con 0x7f543c1430c0 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1724984592 0 0) 0x7f543c0b28a0 con 0x7f543c0a98f0 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 >> v1:172.21.15.193:6789/0 conn(0x7f543c13f710 legacy=0x7f543c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 >> v1:172.21.15.71:6789/0 conn(0x7f543c1430c0 legacy=0x7f543c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c0b44c0 con 0x7f543c0a98f0 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:58.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f542c001670 con 0x7f543c0a98f0 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f543a7fc640 10 monclient: dump: 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f5441ed6640 5 monclient: authenticate success, global_id 14684 2024-03-25T19:50:58.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.344+0000 7f5441ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:58.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f543a7fc640 1 -- 172.21.15.71:0/2258210079 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f542c0017d0 con 0x7f543c0a98f0 2024-03-25T19:50:58.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f543a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:58.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f5441ed6640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:58.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f5441ed6640 4 set_mon_vals no callback set 2024-03-25T19:50:58.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f5441ed6640 10 monclient: shutdown 2024-03-25T19:50:58.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f5441ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f5441ed6640 1 -- 172.21.15.71:0/2258210079 >> v1:172.21.15.71:6790/0 conn(0x7f543c0a98f0 legacy=0x7f543c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f5441ed6640 1 -- 172.21.15.71:0/2258210079 shutdown_connections 2024-03-25T19:50:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.348+0000 7f5441ed6640 1 -- 172.21.15.71:0/2258210079 >> 172.21.15.71:0/2258210079 conn(0x7f543c0b7ea0 msgr2=0x7f543c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.349+0000 7f5441ed6640 1 -- 172.21.15.71:0/2258210079 shutdown_connections 2024-03-25T19:50:58.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.349+0000 7f5441ed6640 1 -- 172.21.15.71:0/2258210079 wait complete. 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.349+0000 7f5441ed6640 10 monclient: build_initial_monmap 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.349+0000 7f5441ed6640 10 monclient: monmap: 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:50:58.350 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:50:58.351 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:58.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.350+0000 7f5441ed6640 1 librados: starting msgr at 2024-03-25T19:50:58.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.350+0000 7f5441ed6640 1 librados: starting objecter 2024-03-25T19:50:58.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.350+0000 7f5441ed6640 1 Processor -- start 2024-03-25T19:50:58.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.350+0000 7f5441ed6640 1 -- start start 2024-03-25T19:50:58.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.350+0000 7f5441ed6640 1 librados: setting wanted keys 2024-03-25T19:50:58.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.350+0000 7f5441ed6640 1 librados: calling monclient init 2024-03-25T19:50:58.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.350+0000 7f5441ed6640 10 monclient: init 2024-03-25T19:50:58.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:58.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:58.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 10 monclient(hunting): picked mon.noname-b con 0x7f543c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:50:58.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 10 monclient(hunting): picked mon.noname-c con 0x7f543c13f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:50:58.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 10 monclient(hunting): picked mon.noname-a con 0x7f543c1430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:50:58.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c1da8f0 con 0x7f543c1430c0 2024-03-25T19:50:58.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c1db770 con 0x7f543c0a98f0 2024-03-25T19:50:58.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c1dab40 con 0x7f543c13f710 2024-03-25T19:50:58.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:58.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.351+0000 7f5441ed6640 10 monclient(hunting): authenticate will time out at 5464.663506s 2024-03-25T19:50:58.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.372+0000 7f543b7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f543c0a98f0 0x7f543c1db060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:58398/0 (socket says 172.21.15.71:58398) 2024-03-25T19:50:58.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.372+0000 7f543b7fe640 1 -- 172.21.15.71:0/1655308786 learned_addr learned my addr 172.21.15.71:0/1655308786 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:50:58.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.373+0000 7f5423fff640 10 client.?.objecter ms_handle_connect 0x7f543c0a98f0 2024-03-25T19:50:58.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.373+0000 7f5423fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:58.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.379+0000 7f5423fff640 10 client.?.objecter ms_handle_connect 0x7f543c1430c0 2024-03-25T19:50:58.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.379+0000 7f5423fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:58.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.380+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677046616 0 0) 0x7f543c1db770 con 0x7f543c0a98f0 2024-03-25T19:50:58.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.380+0000 7f5423fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:58.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.380+0000 7f5423fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:58.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.380+0000 7f5423fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:58.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.380+0000 7f5423fff640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:50:58.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.380+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5428004eb0 con 0x7f543c0a98f0 2024-03-25T19:50:58.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.381+0000 7f5423fff640 10 client.?.objecter ms_handle_connect 0x7f543c13f710 2024-03-25T19:50:58.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.381+0000 7f5423fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:58.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.398+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322113684 0 0) 0x7f543c1dab40 con 0x7f543c13f710 2024-03-25T19:50:58.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.398+0000 7f5423fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:58.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.398+0000 7f5423fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:58.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.398+0000 7f5423fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:58.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.398+0000 7f5423fff640 10 monclient(hunting): my global_id is 14687 2024-03-25T19:50:58.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.398+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c1db770 con 0x7f543c13f710 2024-03-25T19:50:58.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.403+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3502562798 0 0) 0x7f5428004eb0 con 0x7f543c0a98f0 2024-03-25T19:50:58.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.403+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f543c1dab40 con 0x7f543c0a98f0 2024-03-25T19:50:58.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.403+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f542c001d40 con 0x7f543c0a98f0 2024-03-25T19:50:58.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.403+0000 7f5423fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:58.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.403+0000 7f5423fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:58.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.403+0000 7f5423fff640 10 monclient(hunting): dump: 2024-03-25T19:50:58.403 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:58.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.403+0000 7f5423fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:58.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.405+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763088255 0 0) 0x7f543c1da8f0 con 0x7f543c1430c0 2024-03-25T19:50:58.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.405+0000 7f5423fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:58.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.405+0000 7f5423fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:58.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.405+0000 7f5423fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:58.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.405+0000 7f5423fff640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:50:58.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.405+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5428004eb0 con 0x7f543c1430c0 2024-03-25T19:50:58.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.416+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1090656145 0 0) 0x7f543c1db770 con 0x7f543c13f710 2024-03-25T19:50:58.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.416+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f543c1da8f0 con 0x7f543c13f710 2024-03-25T19:50:58.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.416+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f5430001950 con 0x7f543c13f710 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.416+0000 7f5423fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.416+0000 7f5423fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.416+0000 7f5423fff640 10 monclient(hunting): dump: 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:58.416 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:58.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3355710617 0 0) 0x7f543c1dab40 con 0x7f543c0a98f0 2024-03-25T19:50:58.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 >> v1:172.21.15.71:6790/0 conn(0x7f543c13f710 legacy=0x7f543c138530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:58.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 >> v1:172.21.15.71:6789/0 conn(0x7f543c1430c0 legacy=0x7f543c1e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:58.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 1 monclient: found mon.b 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c1390e0 con 0x7f543c0a98f0 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 10 monclient: _finish_auth 0 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5423fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 5 monclient: authenticate success, global_id 4696 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 10 monclient: _renew_subs 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c137c00 con 0x7f543c0a98f0 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 10 monclient: _renew_subs 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c138e20 con 0x7f543c0a98f0 2024-03-25T19:50:58.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.419+0000 7f5441ed6640 1 librados: init done 2024-03-25T19:50:58.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.420+0000 7f5441ed6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:58.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.420+0000 7f5441ed6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:58.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.420+0000 7f5441ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:58.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.420+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c1dacd0 con 0x7f543c0a98f0 2024-03-25T19:50:58.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.698+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f542c004a90 con 0x7f543c0a98f0 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.698+0000 7f5423fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.698+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f542c004e60 con 0x7f543c0a98f0 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.698+0000 7f5423fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.698+0000 7f5423fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.698+0000 7f5423fff640 10 monclient: dump: 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:58.698 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:50:58.699 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:50:58.699 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:50:58.699 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:50:58.699 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:50:58.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.698+0000 7f5439ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:58.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.699+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f542c005080 con 0x7f543c0a98f0 2024-03-25T19:50:58.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.699+0000 7f543affd640 10 monclient: get_auth_request con 0x7f542806dfb0 auth_method 0 2024-03-25T19:50:58.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.700+0000 7f5423fff640 10 client.4696.objecter ms_handle_connect 0x7f542806dfb0 2024-03-25T19:50:58.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.703+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7f542c054930 con 0x7f543c0a98f0 2024-03-25T19:50:58.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.703+0000 7f5423fff640 10 client.4696.objecter ms_dispatch 0x7f543c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:50:58.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.703+0000 7f5423fff640 3 client.4696.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:50:58.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.703+0000 7f5423fff640 3 client.4696.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:50:58.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.703+0000 7f5423fff640 20 client.4696.objecter dump_active .. 0 homeless 2024-03-25T19:50:58.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.704+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f542c050440 con 0x7f543c0a98f0 2024-03-25T19:50:58.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.704+0000 7f5423fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:58.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.704+0000 7f5423fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:58.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.955+0000 7f5441ed6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:58.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.955+0000 7f5441ed6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:58.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.955+0000 7f5441ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:50:58.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:58.955+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f543c0627b0 con 0x7f543c0a98f0 2024-03-25T19:50:59.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.001+0000 7f5423fff640 1 -- 172.21.15.71:0/1655308786 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7f542c052370 con 0x7f543c0a98f0 2024-03-25T19:50:59.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.001+0000 7f5423fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:59.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.001+0000 7f5423fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:59.001 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:50:59.001 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:59.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 10 librados: watch_flush enter 2024-03-25T19:50:59.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 10 librados: watch_flush exit 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 20 client.4696.objecter shutdown clearing up homeless session... 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 10 client.4696.objecter successfully canceled tick 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 >> v1:172.21.15.71:6800/3911735624 conn(0x7f542806dfb0 legacy=0x7f5428070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 10 monclient: shutdown 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 >> v1:172.21.15.193:6789/0 conn(0x7f543c0a98f0 legacy=0x7f543c1db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 shutdown_connections 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.003+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 >> 172.21.15.71:0/1655308786 conn(0x7f543c0b7ea0 msgr2=0x7f543c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.004+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 shutdown_connections 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.004+0000 7f5441ed6640 1 -- 172.21.15.71:0/1655308786 wait complete. 2024-03-25T19:50:59.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:50:59.004+0000 7f5441ed6640 1 librados: shutdown 2024-03-25T19:50:59.015 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:05.821 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:05.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.925+0000 7f6650cd9640 10 monclient: get_monmap_and_config 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.925+0000 7f6650cd9640 10 monclient: build_initial_monmap 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.925+0000 7f6650cd9640 10 monclient: monmap: 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:51:05.926060+0000 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:51:05.926060+0000 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:51:05.925 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.925+0000 7f6650cd9640 1 Processor -- start 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.925+0000 7f6650cd9640 1 -- start start 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.925+0000 7f6650cd9640 10 monclient: init 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f664c06c380 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f664c06d670 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f664c0af8f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c06c9a0 con 0x7f664c06c380 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0ada90 con 0x7f664c06d670 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0b0d10 con 0x7f664c0af8f0 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.926+0000 7f6650cd9640 10 monclient(hunting): authenticate will time out at 5472.238375s 2024-03-25T19:51:05.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.927+0000 7f6649d9b640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f664c06d670 0x7f664c0ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:34996/0 (socket says 172.21.15.71:34996) 2024-03-25T19:51:05.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.927+0000 7f6649d9b640 1 -- 172.21.15.71:0/3280194613 learned_addr learned my addr 172.21.15.71:0/3280194613 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:05.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.975+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183792458 0 0) 0x7f664c0b0d10 con 0x7f664c0af8f0 2024-03-25T19:51:05.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.975+0000 7f664959a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:05.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.975+0000 7f664959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:05.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.975+0000 7f664959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:05.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.975+0000 7f664959a640 10 monclient(hunting): my global_id is 14690 2024-03-25T19:51:05.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.975+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634003520 con 0x7f664c0af8f0 2024-03-25T19:51:05.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3880461653 0 0) 0x7f6634003520 con 0x7f664c0af8f0 2024-03-25T19:51:05.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 >> v1:172.21.15.193:6789/0 conn(0x7f664c06d670 legacy=0x7f664c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:05.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 >> v1:172.21.15.71:6789/0 conn(0x7f664c06c380 legacy=0x7f664c0b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 10 monclient: _finish_hunting 0 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 1 monclient: found mon.noname-b 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0add00 con 0x7f664c0af8f0 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 10 monclient: _finish_auth 0 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 10 monclient: _check_auth_tickets 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6644001a10 con 0x7f664c0af8f0 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f664959a640 10 monclient: dump: 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:05.983 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f6650cd9640 5 monclient: authenticate success, global_id 14690 2024-03-25T19:51:05.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.982+0000 7f6650cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:05.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.996+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6644001b70 con 0x7f664c0af8f0 2024-03-25T19:51:05.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.996+0000 7f664959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:05.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.996+0000 7f664959a640 1 -- 172.21.15.71:0/3280194613 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6644001f40 con 0x7f664c0af8f0 2024-03-25T19:51:05.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.996+0000 7f664959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:05.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.997+0000 7f664959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.997+0000 7f664959a640 10 monclient: dump: 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.997+0000 7f6650cd9640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:05.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.997+0000 7f6650cd9640 4 set_mon_vals no callback set 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.997+0000 7f6650cd9640 10 monclient: shutdown 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.997+0000 7f6650cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.997+0000 7f6650cd9640 1 -- 172.21.15.71:0/3280194613 >> v1:172.21.15.71:6790/0 conn(0x7f664c0af8f0 legacy=0x7f664c0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.998+0000 7f6650cd9640 1 -- 172.21.15.71:0/3280194613 shutdown_connections 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.998+0000 7f6650cd9640 1 -- 172.21.15.71:0/3280194613 >> 172.21.15.71:0/3280194613 conn(0x7f664c0b5c00 msgr2=0x7f664c0b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.998+0000 7f6650cd9640 1 -- 172.21.15.71:0/3280194613 shutdown_connections 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.998+0000 7f6650cd9640 1 -- 172.21.15.71:0/3280194613 wait complete. 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 10 monclient: build_initial_monmap 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 10 monclient: monmap: 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:05.999 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 1 librados: starting msgr at 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 1 librados: starting objecter 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 1 Processor -- start 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 1 -- start start 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 1 librados: setting wanted keys 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 1 librados: calling monclient init 2024-03-25T19:51:06.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:05.999+0000 7f6650cd9640 10 monclient: init 2024-03-25T19:51:06.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:06.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:06.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f664c06c380 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:06.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f664c06d670 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:06.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f664c0af8f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:06.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c143d70 con 0x7f664c06d670 2024-03-25T19:51:06.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c13b960 con 0x7f664c0af8f0 2024-03-25T19:51:06.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c145340 con 0x7f664c06c380 2024-03-25T19:51:06.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:06.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.000+0000 7f6650cd9640 10 monclient(hunting): authenticate will time out at 5472.312311s 2024-03-25T19:51:06.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.027+0000 7f664ad9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f664c0af8f0 0x7f664c13b250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:35048/0 (socket says 172.21.15.71:35048) 2024-03-25T19:51:06.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.027+0000 7f664ad9d640 1 -- 172.21.15.71:0/850094371 learned_addr learned my addr 172.21.15.71:0/850094371 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:06.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.028+0000 7f663affd640 10 client.?.objecter ms_handle_connect 0x7f664c0af8f0 2024-03-25T19:51:06.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.028+0000 7f663affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:06.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.031+0000 7f663affd640 10 client.?.objecter ms_handle_connect 0x7f664c06c380 2024-03-25T19:51:06.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.031+0000 7f663affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:06.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.037+0000 7f663affd640 10 client.?.objecter ms_handle_connect 0x7f664c06d670 2024-03-25T19:51:06.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.037+0000 7f663affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:06.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.041+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826960316 0 0) 0x7f664c13b960 con 0x7f664c0af8f0 2024-03-25T19:51:06.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.041+0000 7f663affd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:06.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.041+0000 7f663affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:06.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.041+0000 7f663affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:06.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.041+0000 7f663affd640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:51:06.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.041+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f662c004eb0 con 0x7f664c0af8f0 2024-03-25T19:51:06.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.049+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231100544 0 0) 0x7f664c143d70 con 0x7f664c06d670 2024-03-25T19:51:06.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.049+0000 7f663affd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:06.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.049+0000 7f663affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:06.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.049+0000 7f663affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:06.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.049+0000 7f663affd640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:51:06.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.050+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c13b960 con 0x7f664c06d670 2024-03-25T19:51:06.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.050+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981460823 0 0) 0x7f664c145340 con 0x7f664c06c380 2024-03-25T19:51:06.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.050+0000 7f663affd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:06.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.050+0000 7f663affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:06.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.050+0000 7f663affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:06.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.050+0000 7f663affd640 10 monclient(hunting): my global_id is 14693 2024-03-25T19:51:06.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.050+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c143d70 con 0x7f664c06c380 2024-03-25T19:51:06.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.055+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2233401532 0 0) 0x7f662c004eb0 con 0x7f664c0af8f0 2024-03-25T19:51:06.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.055+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f664c145340 con 0x7f664c0af8f0 2024-03-25T19:51:06.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.055+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6644001ec0 con 0x7f664c0af8f0 2024-03-25T19:51:06.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.055+0000 7f663affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:06.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.055+0000 7f663affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:06.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.055+0000 7f663affd640 10 monclient(hunting): dump: 2024-03-25T19:51:06.055 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:06.055 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:06.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.055+0000 7f663affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:06.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.064+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 434346434 0 0) 0x7f664c145340 con 0x7f664c0af8f0 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.064+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 >> v1:172.21.15.71:6790/0 conn(0x7f664c06c380 legacy=0x7f664c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.064+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 >> v1:172.21.15.71:6789/0 conn(0x7f664c06d670 legacy=0x7f664c13ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.064+0000 7f663affd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.064+0000 7f663affd640 1 monclient: found mon.b 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.064+0000 7f663affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.064+0000 7f663affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.064+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c1db560 con 0x7f664c0af8f0 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f663affd640 10 monclient: _finish_auth 0 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f663affd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f663affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 5 monclient: authenticate success, global_id 4702 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 10 monclient: _renew_subs 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:06.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c1485b0 con 0x7f664c0af8f0 2024-03-25T19:51:06.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 10 client.4702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:06.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 10 monclient: _renew_subs 2024-03-25T19:51:06.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:06.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c1439a0 con 0x7f664c0af8f0 2024-03-25T19:51:06.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.065+0000 7f6650cd9640 1 librados: init done 2024-03-25T19:51:06.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.066+0000 7f6650cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:06.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.066+0000 7f6650cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:06.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.066+0000 7f6650cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:06.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.066+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c0627b0 con 0x7f664c0af8f0 2024-03-25T19:51:06.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.355+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6644003c20 con 0x7f664c0af8f0 2024-03-25T19:51:06.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.355+0000 7f663affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:06.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.355+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f6644002100 con 0x7f664c0af8f0 2024-03-25T19:51:06.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.355+0000 7f663affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:06.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.355+0000 7f663affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:06.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.355+0000 7f663affd640 10 monclient: dump: 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:06.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.355+0000 7f6648d99640 4 set_mon_vals no callback set 2024-03-25T19:51:06.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.356+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f6644004070 con 0x7f664c0af8f0 2024-03-25T19:51:06.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.357+0000 7f664a59c640 10 monclient: get_auth_request con 0x7f662c06ddf0 auth_method 0 2024-03-25T19:51:06.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.357+0000 7f663affd640 10 client.4702.objecter ms_handle_connect 0x7f662c06ddf0 2024-03-25T19:51:06.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.361+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7f6644054c50 con 0x7f664c0af8f0 2024-03-25T19:51:06.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.361+0000 7f663affd640 10 client.4702.objecter ms_dispatch 0x7f664c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:06.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.361+0000 7f663affd640 3 client.4702.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:06.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.361+0000 7f663affd640 3 client.4702.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:06.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.361+0000 7f663affd640 20 client.4702.objecter dump_active .. 0 homeless 2024-03-25T19:51:06.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.361+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6644050760 con 0x7f664c0af8f0 2024-03-25T19:51:06.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.361+0000 7f663affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:06.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.361+0000 7f663affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:06.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.609+0000 7f6650cd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:06.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.609+0000 7f6650cd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:06.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.609+0000 7f6650cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:06.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.609+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f664c070120 con 0x7f664c0af8f0 2024-03-25T19:51:06.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.633+0000 7f663affd640 1 -- 172.21.15.71:0/850094371 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7f6644052690 con 0x7f664c0af8f0 2024-03-25T19:51:06.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.633+0000 7f663affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:06.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.633+0000 7f663affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:06.633 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:51:06.633 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 10 librados: watch_flush enter 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 10 librados: watch_flush exit 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 20 client.4702.objecter shutdown clearing up homeless session... 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 10 client.4702.objecter successfully canceled tick 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 >> v1:172.21.15.71:6800/3911735624 conn(0x7f662c06ddf0 legacy=0x7f662c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 10 monclient: shutdown 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 >> v1:172.21.15.193:6789/0 conn(0x7f664c0af8f0 legacy=0x7f664c13b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 shutdown_connections 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.635+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 >> 172.21.15.71:0/850094371 conn(0x7f664c0b5c00 msgr2=0x7f664c13fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:06.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.636+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 shutdown_connections 2024-03-25T19:51:06.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.636+0000 7f6650cd9640 1 -- 172.21.15.71:0/850094371 wait complete. 2024-03-25T19:51:06.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:06.636+0000 7f6650cd9640 1 librados: shutdown 2024-03-25T19:51:06.647 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:08.313 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:51.070 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.325 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:51.082 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.365 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:10:51.122 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.370 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:51.089 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.375 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:51.094 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.376 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:51.133 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.381 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:51.137 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.412 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:10:51.168 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.415 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:51.134 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:13.455 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:13.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.573+0000 7fc0df264640 10 monclient: get_monmap_and_config 2024-03-25T19:51:13.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.573+0000 7fc0df264640 10 monclient: build_initial_monmap 2024-03-25T19:51:13.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.573+0000 7fc0df264640 10 monclient: monmap: 2024-03-25T19:51:13.573 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:13.573 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:13.573 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:51:13.574419+0000 2024-03-25T19:51:13.574 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:51:13.574419+0000 2024-03-25T19:51:13.574 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:13.574 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:13.574 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:13.574 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:51:13.574 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:51:13.574 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.574+0000 7fc0df264640 1 Processor -- start 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.574+0000 7fc0df264640 1 -- start start 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.574+0000 7fc0df264640 10 monclient: init 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.575+0000 7fc0df264640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.575+0000 7fc0df264640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.575+0000 7fc0df264640 10 monclient(hunting): picked mon.noname-c con 0x7fc0d806c380 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.575+0000 7fc0df264640 10 monclient(hunting): picked mon.noname-b con 0x7fc0d806d670 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.575+0000 7fc0df264640 10 monclient(hunting): picked mon.noname-a con 0x7fc0d80af8f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.576+0000 7fc0df264640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d806c9a0 con 0x7fc0d80af8f0 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.576+0000 7fc0df264640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80ada90 con 0x7fc0d806c380 2024-03-25T19:51:13.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.576+0000 7fc0df264640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80b0d10 con 0x7fc0d806d670 2024-03-25T19:51:13.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.576+0000 7fc0df264640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:13.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.576+0000 7fc0df264640 10 monclient(hunting): authenticate will time out at 5479.888171s 2024-03-25T19:51:13.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.583+0000 7fc0dd000640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc0d806c380 0x7fc0d80b0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:42406/0 (socket says 172.21.15.71:42406) 2024-03-25T19:51:13.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.583+0000 7fc0dd000640 1 -- 172.21.15.71:0/3158172306 learned_addr learned my addr 172.21.15.71:0/3158172306 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:13.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.598+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138879503 0 0) 0x7fc0d80b0d10 con 0x7fc0d806d670 2024-03-25T19:51:13.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.598+0000 7fc0d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:13.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.598+0000 7fc0d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:13.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.598+0000 7fc0d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:13.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.598+0000 7fc0d77fe640 10 monclient(hunting): my global_id is 14696 2024-03-25T19:51:13.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.598+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c8003520 con 0x7fc0d806d670 2024-03-25T19:51:13.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.600+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652363011 0 0) 0x7fc0d806c9a0 con 0x7fc0d80af8f0 2024-03-25T19:51:13.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.600+0000 7fc0d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.600+0000 7fc0d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.600+0000 7fc0d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.600+0000 7fc0d77fe640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.601+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d80b0d10 con 0x7fc0d80af8f0 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.601+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864437880 0 0) 0x7fc0d80ada90 con 0x7fc0d806c380 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.601+0000 7fc0d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.601+0000 7fc0d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.601+0000 7fc0d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.601+0000 7fc0d77fe640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:51:13.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.601+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d806c9a0 con 0x7fc0d806c380 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4260637977 0 0) 0x7fc0d806c9a0 con 0x7fc0d806c380 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 >> v1:172.21.15.71:6790/0 conn(0x7fc0d806d670 legacy=0x7fc0d80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 >> v1:172.21.15.71:6789/0 conn(0x7fc0d80af8f0 legacy=0x7fc0d80aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 1 monclient: found mon.noname-c 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80add00 con 0x7fc0d806c380 2024-03-25T19:51:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc0cc001930 con 0x7fc0d806c380 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0d77fe640 10 monclient: dump: 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0df264640 5 monclient: authenticate success, global_id 4705 2024-03-25T19:51:13.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.610+0000 7fc0df264640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.626+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0cc001a90 con 0x7fc0d806c380 2024-03-25T19:51:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.626+0000 7fc0d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.626+0000 7fc0d77fe640 1 -- 172.21.15.71:0/3158172306 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc0cc001ee0 con 0x7fc0d806c380 2024-03-25T19:51:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.626+0000 7fc0d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.626+0000 7fc0d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:13.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.626+0000 7fc0d77fe640 10 monclient: dump: 2024-03-25T19:51:13.626 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:13.626 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.626+0000 7fc0df264640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:13.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.626+0000 7fc0df264640 4 set_mon_vals no callback set 2024-03-25T19:51:13.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.627+0000 7fc0df264640 10 monclient: shutdown 2024-03-25T19:51:13.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.627+0000 7fc0df264640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:13.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.627+0000 7fc0df264640 1 -- 172.21.15.71:0/3158172306 >> v1:172.21.15.193:6789/0 conn(0x7fc0d806c380 legacy=0x7fc0d80b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.627+0000 7fc0df264640 1 -- 172.21.15.71:0/3158172306 shutdown_connections 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.627+0000 7fc0df264640 1 -- 172.21.15.71:0/3158172306 >> 172.21.15.71:0/3158172306 conn(0x7fc0d80b5c00 msgr2=0x7fc0d80b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.627+0000 7fc0df264640 1 -- 172.21.15.71:0/3158172306 shutdown_connections 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.627+0000 7fc0df264640 1 -- 172.21.15.71:0/3158172306 wait complete. 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 10 monclient: build_initial_monmap 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 10 monclient: monmap: 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:51:13.630 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 1 librados: starting msgr at 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 1 librados: starting objecter 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 1 Processor -- start 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 1 -- start start 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 1 librados: setting wanted keys 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 1 librados: calling monclient init 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.628+0000 7fc0df264640 10 monclient: init 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 10 monclient(hunting): picked mon.noname-b con 0x7fc0d806c380 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 10 monclient(hunting): picked mon.noname-c con 0x7fc0d806d670 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 10 monclient(hunting): picked mon.noname-a con 0x7fc0d80af8f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d813faa0 con 0x7fc0d80af8f0 2024-03-25T19:51:13.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d8141f20 con 0x7fc0d806c380 2024-03-25T19:51:13.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d8149570 con 0x7fc0d806d670 2024-03-25T19:51:13.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:13.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.629+0000 7fc0df264640 10 monclient(hunting): authenticate will time out at 5479.941641s 2024-03-25T19:51:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.647+0000 7fc0dd000640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc0d806c380 0x7fc0d813ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:42458/0 (socket says 172.21.15.71:42458) 2024-03-25T19:51:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.647+0000 7fc0dd000640 1 -- 172.21.15.71:0/3196489117 learned_addr learned my addr 172.21.15.71:0/3196489117 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.648+0000 7fc0d57fa640 10 client.?.objecter ms_handle_connect 0x7fc0d806c380 2024-03-25T19:51:13.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.648+0000 7fc0d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:13.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.649+0000 7fc0d57fa640 10 client.?.objecter ms_handle_connect 0x7fc0d80af8f0 2024-03-25T19:51:13.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.649+0000 7fc0d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:13.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.650+0000 7fc0d57fa640 10 client.?.objecter ms_handle_connect 0x7fc0d806d670 2024-03-25T19:51:13.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.650+0000 7fc0d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:13.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.657+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96151054 0 0) 0x7fc0d8149570 con 0x7fc0d806d670 2024-03-25T19:51:13.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.657+0000 7fc0d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:13.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.657+0000 7fc0d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:13.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.657+0000 7fc0d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:13.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.657+0000 7fc0d57fa640 10 monclient(hunting): my global_id is 14699 2024-03-25T19:51:13.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.657+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc004eb0 con 0x7fc0d806d670 2024-03-25T19:51:13.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.659+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699731473 0 0) 0x7fc0d8141f20 con 0x7fc0d806c380 2024-03-25T19:51:13.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.659+0000 7fc0d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:13.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.659+0000 7fc0d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:13.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.659+0000 7fc0d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:13.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.659+0000 7fc0d57fa640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:51:13.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.659+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d8149570 con 0x7fc0d806c380 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.670+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3564725296 0 0) 0x7fc0bc004eb0 con 0x7fc0d806d670 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.670+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0d8141f20 con 0x7fc0d806d670 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.670+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc0c4001950 con 0x7fc0d806d670 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.670+0000 7fc0d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.670+0000 7fc0d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.670+0000 7fc0d57fa640 10 monclient(hunting): dump: 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:13.670 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:13.671 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:13.671 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:13.671 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:13.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.670+0000 7fc0d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:13.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.673+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775043453 0 0) 0x7fc0d813faa0 con 0x7fc0d80af8f0 2024-03-25T19:51:13.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.673+0000 7fc0d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:13.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.673+0000 7fc0d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:13.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.673+0000 7fc0d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:13.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.673+0000 7fc0d57fa640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:51:13.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.673+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc004eb0 con 0x7fc0d80af8f0 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1966144556 0 0) 0x7fc0d8141f20 con 0x7fc0d806d670 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 >> v1:172.21.15.193:6789/0 conn(0x7fc0d806c380 legacy=0x7fc0d813ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 >> v1:172.21.15.71:6789/0 conn(0x7fc0d80af8f0 legacy=0x7fc0d813c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 1 monclient: found mon.c 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d8140cd0 con 0x7fc0d806d670 2024-03-25T19:51:13.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 5 monclient: authenticate success, global_id 14699 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 10 monclient: _renew_subs 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d813d0e0 con 0x7fc0d806d670 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 10 client.14699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 10 monclient: _renew_subs 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d8144740 con 0x7fc0d806d670 2024-03-25T19:51:13.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.684+0000 7fc0df264640 1 librados: init done 2024-03-25T19:51:13.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.685+0000 7fc0df264640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:13.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.685+0000 7fc0df264640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:13.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.685+0000 7fc0df264640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:51:13.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.685+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d8140ad0 con 0x7fc0d806d670 2024-03-25T19:51:13.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.989+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0c4003610 con 0x7fc0d806d670 2024-03-25T19:51:13.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.989+0000 7fc0d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:13.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.989+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fc0c4001d80 con 0x7fc0d806d670 2024-03-25T19:51:13.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.989+0000 7fc0d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:13.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.989+0000 7fc0d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.989+0000 7fc0d57fa640 10 monclient: dump: 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:13.990 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.989+0000 7fc0d6ffd640 4 set_mon_vals no callback set 2024-03-25T19:51:13.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.990+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fc0c4003ae0 con 0x7fc0d806d670 2024-03-25T19:51:13.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.991+0000 7fc0dd000640 10 monclient: get_auth_request con 0x7fc0bc06de00 auth_method 0 2024-03-25T19:51:13.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.991+0000 7fc0d57fa640 10 client.14699.objecter ms_handle_connect 0x7fc0bc06de00 2024-03-25T19:51:13.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.996+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7fc0c4053550 con 0x7fc0d806d670 2024-03-25T19:51:13.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.996+0000 7fc0d57fa640 10 client.14699.objecter ms_dispatch 0x7fc0d8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:13.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.996+0000 7fc0d57fa640 3 client.14699.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:13.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.996+0000 7fc0d57fa640 3 client.14699.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:13.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.996+0000 7fc0d57fa640 20 client.14699.objecter dump_active .. 0 homeless 2024-03-25T19:51:13.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.996+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0c404f060 con 0x7fc0d806d670 2024-03-25T19:51:13.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.996+0000 7fc0d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:13.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:13.996+0000 7fc0d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:14.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.223+0000 7fc0df264640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:14.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.223+0000 7fc0df264640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:14.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.223+0000 7fc0df264640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-25T19:51:14.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.223+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0d8140e60 con 0x7fc0d806d670 2024-03-25T19:51:14.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.259+0000 7fc0d57fa640 1 -- 172.21.15.71:0/3196489117 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7fc0c4050f90 con 0x7fc0d806d670 2024-03-25T19:51:14.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.259+0000 7fc0d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:14.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.259+0000 7fc0d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:14.259 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:51:14.259 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:14.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.261+0000 7fc0df264640 10 librados: watch_flush enter 2024-03-25T19:51:14.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 10 librados: watch_flush exit 2024-03-25T19:51:14.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 20 client.14699.objecter shutdown clearing up homeless session... 2024-03-25T19:51:14.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 10 client.14699.objecter successfully canceled tick 2024-03-25T19:51:14.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 >> v1:172.21.15.71:6800/3911735624 conn(0x7fc0bc06de00 legacy=0x7fc0bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:14.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 10 monclient: shutdown 2024-03-25T19:51:14.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:14.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 >> v1:172.21.15.71:6790/0 conn(0x7fc0d806d670 legacy=0x7fc0d813f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:14.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 shutdown_connections 2024-03-25T19:51:14.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 >> 172.21.15.71:0/3196489117 conn(0x7fc0d80b5c00 msgr2=0x7fc0d8148460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:14.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 shutdown_connections 2024-03-25T19:51:14.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 1 -- 172.21.15.71:0/3196489117 wait complete. 2024-03-25T19:51:14.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:14.262+0000 7fc0df264640 1 librados: shutdown 2024-03-25T19:51:14.273 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:21.080 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:21.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.193+0000 7fd412932640 10 monclient: get_monmap_and_config 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.193+0000 7fd412932640 10 monclient: build_initial_monmap 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.193+0000 7fd412932640 10 monclient: monmap: 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:51:21.193964+0000 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:51:21.193964+0000 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:51:21.193 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.193+0000 7fd412932640 1 Processor -- start 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 1 -- start start 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 10 monclient: init 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 10 monclient(hunting): picked mon.noname-b con 0x7fd40c0a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 10 monclient(hunting): picked mon.noname-c con 0x7fd40c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c1430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c142520 con 0x7fd40c1430c0 2024-03-25T19:51:21.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0b28a0 con 0x7fd40c13f710 2024-03-25T19:51:21.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0b2a30 con 0x7fd40c0a98f0 2024-03-25T19:51:21.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:21.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.194+0000 7fd412932640 10 monclient(hunting): authenticate will time out at 5487.506735s 2024-03-25T19:51:21.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.206+0000 7fd40bfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd40c0a98f0 0x7fd40c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48926/0 (socket says 172.21.15.71:48926) 2024-03-25T19:51:21.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.206+0000 7fd40bfff640 1 -- 172.21.15.71:0/1987824064 learned_addr learned my addr 172.21.15.71:0/1987824064 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:21.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.219+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078555314 0 0) 0x7fd40c0b2a30 con 0x7fd40c0a98f0 2024-03-25T19:51:21.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.219+0000 7fd40affd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:21.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.219+0000 7fd40affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:21.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.219+0000 7fd40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:21.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.219+0000 7fd40affd640 10 monclient(hunting): my global_id is 14702 2024-03-25T19:51:21.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.219+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f4003520 con 0x7fd40c0a98f0 2024-03-25T19:51:21.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.231+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033440058 0 0) 0x7fd40c0b28a0 con 0x7fd40c13f710 2024-03-25T19:51:21.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.231+0000 7fd40affd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:21.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.231+0000 7fd40affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:21.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.231+0000 7fd40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:21.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.231+0000 7fd40affd640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:51:21.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.231+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40c0b2a30 con 0x7fd40c13f710 2024-03-25T19:51:21.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503955312 0 0) 0x7fd3f4003520 con 0x7fd40c0a98f0 2024-03-25T19:51:21.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 >> v1:172.21.15.193:6789/0 conn(0x7fd40c13f710 legacy=0x7fd40c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 >> v1:172.21.15.71:6789/0 conn(0x7fd40c1430c0 legacy=0x7fd40c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 1 monclient: found mon.noname-b 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0b44c0 con 0x7fd40c0a98f0 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 10 monclient: _finish_auth 0 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd3fc001690 con 0x7fd40c0a98f0 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd412932640 5 monclient: authenticate success, global_id 14702 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd412932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:21.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd40affd640 10 monclient: dump: 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:21.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.238+0000 7fd412932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.249+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3fc0017f0 con 0x7fd40c0a98f0 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.249+0000 7fd40affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.249+0000 7fd40affd640 1 -- 172.21.15.71:0/1987824064 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd3fc001c00 con 0x7fd40c0a98f0 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.249+0000 7fd40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.249+0000 7fd40affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.249+0000 7fd40affd640 10 monclient: dump: 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.249 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:21.250 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:21.250 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:21.250 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:21.250 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:21.250 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:21.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.249+0000 7fd412932640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:21.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.249+0000 7fd412932640 4 set_mon_vals no callback set 2024-03-25T19:51:21.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.250+0000 7fd412932640 10 monclient: shutdown 2024-03-25T19:51:21.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.250+0000 7fd412932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:21.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.250+0000 7fd412932640 1 -- 172.21.15.71:0/1987824064 >> v1:172.21.15.71:6790/0 conn(0x7fd40c0a98f0 legacy=0x7fd40c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.250+0000 7fd412932640 1 -- 172.21.15.71:0/1987824064 shutdown_connections 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.250+0000 7fd412932640 1 -- 172.21.15.71:0/1987824064 >> 172.21.15.71:0/1987824064 conn(0x7fd40c0b7ea0 msgr2=0x7fd40c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.250+0000 7fd412932640 1 -- 172.21.15.71:0/1987824064 shutdown_connections 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.250+0000 7fd412932640 1 -- 172.21.15.71:0/1987824064 wait complete. 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.251+0000 7fd412932640 10 monclient: build_initial_monmap 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.251+0000 7fd412932640 10 monclient: monmap: 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:21.252 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.251+0000 7fd412932640 1 librados: starting msgr at 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.251+0000 7fd412932640 1 librados: starting objecter 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.251+0000 7fd412932640 1 Processor -- start 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 1 -- start start 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 1 librados: setting wanted keys 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 1 librados: calling monclient init 2024-03-25T19:51:21.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 10 monclient: init 2024-03-25T19:51:21.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:21.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:21.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c0a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:21.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 10 monclient(hunting): picked mon.noname-b con 0x7fd40c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:21.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 10 monclient(hunting): picked mon.noname-c con 0x7fd40c1430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:21.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c138b20 con 0x7fd40c0a98f0 2024-03-25T19:51:21.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c137a80 con 0x7fd40c13f710 2024-03-25T19:51:21.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c139880 con 0x7fd40c1430c0 2024-03-25T19:51:21.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:21.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.252+0000 7fd412932640 10 monclient(hunting): authenticate will time out at 5487.564835s 2024-03-25T19:51:21.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.265+0000 7fd40b7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd40c13f710 0x7fd40c139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:33204/0 (socket says 172.21.15.71:33204) 2024-03-25T19:51:21.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.265+0000 7fd40b7fe640 1 -- 172.21.15.71:0/2220716720 learned_addr learned my addr 172.21.15.71:0/2220716720 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:21.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.266+0000 7fd408ff9640 10 client.?.objecter ms_handle_connect 0x7fd40c13f710 2024-03-25T19:51:21.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.266+0000 7fd408ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:21.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.268+0000 7fd408ff9640 10 client.?.objecter ms_handle_connect 0x7fd40c0a98f0 2024-03-25T19:51:21.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.268+0000 7fd408ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:21.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.269+0000 7fd408ff9640 10 client.?.objecter ms_handle_connect 0x7fd40c1430c0 2024-03-25T19:51:21.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.269+0000 7fd408ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:21.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.273+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104529152 0 0) 0x7fd40c137a80 con 0x7fd40c13f710 2024-03-25T19:51:21.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.273+0000 7fd408ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:21.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.273+0000 7fd408ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:21.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.273+0000 7fd408ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:21.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.273+0000 7fd408ff9640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:51:21.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.273+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f8004eb0 con 0x7fd40c13f710 2024-03-25T19:51:21.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.281+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047772810 0 0) 0x7fd40c139880 con 0x7fd40c1430c0 2024-03-25T19:51:21.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.281+0000 7fd408ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:21.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.281+0000 7fd408ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:21.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.281+0000 7fd408ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:21.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.281+0000 7fd408ff9640 10 monclient(hunting): my global_id is 14705 2024-03-25T19:51:21.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.281+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd40c137a80 con 0x7fd40c1430c0 2024-03-25T19:51:21.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.287+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4215903136 0 0) 0x7fd3f8004eb0 con 0x7fd40c13f710 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.288+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd40c139880 con 0x7fd40c13f710 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.288+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd400001a10 con 0x7fd40c13f710 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.288+0000 7fd408ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.288+0000 7fd408ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.288+0000 7fd408ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:21.288 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:21.289 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:21.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.288+0000 7fd408ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:21.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.290+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586056380 0 0) 0x7fd40c138b20 con 0x7fd40c0a98f0 2024-03-25T19:51:21.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.290+0000 7fd408ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:21.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.290+0000 7fd408ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:21.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.290+0000 7fd408ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:21.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.290+0000 7fd408ff9640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:51:21.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.290+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f8004eb0 con 0x7fd40c0a98f0 2024-03-25T19:51:21.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.304+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2014679513 0 0) 0x7fd40c137a80 con 0x7fd40c1430c0 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.304+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd40c138b20 con 0x7fd40c1430c0 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.304+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd404001b00 con 0x7fd40c1430c0 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.304+0000 7fd408ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.304+0000 7fd408ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.304+0000 7fd408ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:21.304 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:21.305 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:21.305 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:21.305 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.307+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4248221791 0 0) 0x7fd3f8004eb0 con 0x7fd40c0a98f0 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.307+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd40c137a80 con 0x7fd40c0a98f0 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.307+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd3fc001b00 con 0x7fd40c0a98f0 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.307+0000 7fd408ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.307+0000 7fd408ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.307+0000 7fd408ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:21.307 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.308 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.308 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:21.308 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:21.308 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:21.308 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:21.308 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:21.308 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1562898542 0 0) 0x7fd40c139880 con 0x7fd40c13f710 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 >> v1:172.21.15.71:6790/0 conn(0x7fd40c1430c0 legacy=0x7fd40c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 >> v1:172.21.15.71:6789/0 conn(0x7fd40c0a98f0 legacy=0x7fd40c137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 1 monclient: found mon.b 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c139a10 con 0x7fd40c13f710 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:21.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd408ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 5 monclient: authenticate success, global_id 4714 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 10 monclient: _renew_subs 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c1dab00 con 0x7fd40c13f710 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 10 monclient: _renew_subs 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c13b310 con 0x7fd40c13f710 2024-03-25T19:51:21.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.313+0000 7fd412932640 1 librados: init done 2024-03-25T19:51:21.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.314+0000 7fd412932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:21.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.314+0000 7fd412932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:21.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.314+0000 7fd412932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:21.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.314+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c074cc0 con 0x7fd40c13f710 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.632+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd400001c50 con 0x7fd40c13f710 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.632+0000 7fd408ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.632+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fd400003a60 con 0x7fd40c13f710 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.632+0000 7fd408ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.632+0000 7fd408ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.632+0000 7fd40a7fc640 4 set_mon_vals no callback set 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.632+0000 7fd408ff9640 10 monclient: dump: 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:21.632 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.633 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:21.633 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:21.633 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:21.633 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:21.633 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:21.633 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:21.633 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:21.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.633+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fd400003c80 con 0x7fd40c13f710 2024-03-25T19:51:21.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.634+0000 7fd40bfff640 10 monclient: get_auth_request con 0x7fd3f806e160 auth_method 0 2024-03-25T19:51:21.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.634+0000 7fd408ff9640 10 client.4714.objecter ms_handle_connect 0x7fd3f806e160 2024-03-25T19:51:21.637 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.637+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7fd400054800 con 0x7fd40c13f710 2024-03-25T19:51:21.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.637+0000 7fd408ff9640 10 client.4714.objecter ms_dispatch 0x7fd40c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:21.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.637+0000 7fd408ff9640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:21.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.637+0000 7fd408ff9640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:21.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.637+0000 7fd408ff9640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T19:51:21.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.638+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd400050310 con 0x7fd40c13f710 2024-03-25T19:51:21.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.638+0000 7fd408ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:21.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.638+0000 7fd408ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:21.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.883+0000 7fd412932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:21.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.883+0000 7fd412932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:21.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.883+0000 7fd412932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:21.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:21.883+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd40c0b3e80 con 0x7fd40c13f710 2024-03-25T19:51:22.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.000+0000 7fd408ff9640 1 -- 172.21.15.71:0/2220716720 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7fd400052240 con 0x7fd40c13f710 2024-03-25T19:51:22.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.000+0000 7fd408ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:22.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.000+0000 7fd408ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:22.000 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:51:22.000 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:22.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.002+0000 7fd412932640 10 librados: watch_flush enter 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.002+0000 7fd412932640 10 librados: watch_flush exit 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.002+0000 7fd412932640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.002+0000 7fd412932640 10 client.4714.objecter successfully canceled tick 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.002+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 >> v1:172.21.15.71:6800/3911735624 conn(0x7fd3f806e160 legacy=0x7fd3f8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.002+0000 7fd412932640 10 monclient: shutdown 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.002+0000 7fd412932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.002+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 >> v1:172.21.15.193:6789/0 conn(0x7fd40c13f710 legacy=0x7fd40c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.003+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 shutdown_connections 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.003+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 >> 172.21.15.71:0/2220716720 conn(0x7fd40c0b7ea0 msgr2=0x7fd40c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.003+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 shutdown_connections 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.003+0000 7fd412932640 1 -- 172.21.15.71:0/2220716720 wait complete. 2024-03-25T19:51:22.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:22.003+0000 7fd412932640 1 librados: shutdown 2024-03-25T19:51:22.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:28.355 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:11.111 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.363 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:11.119 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.395 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:11.151 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.407 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:11.163 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.416 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:11.172 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.420 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:11.139 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.433 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:11.152 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.454 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:11.211 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.460 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:11.179 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:28.820 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:28.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.937+0000 7f3982ae4640 10 monclient: get_monmap_and_config 2024-03-25T19:51:28.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.937+0000 7f3982ae4640 10 monclient: build_initial_monmap 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.938+0000 7f3982ae4640 10 monclient: monmap: 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:51:28.938886+0000 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:51:28.938886+0000 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:51:28.938 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:28.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 1 Processor -- start 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 1 -- start start 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 10 monclient: init 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 10 monclient(hunting): picked mon.noname-a con 0x7f397c13fc60 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 10 monclient(hunting): picked mon.noname-b con 0x7f397c070570 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 10 monclient(hunting): picked mon.noname-c con 0x7f397c0727a0 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.939+0000 7f3982ae4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c13c9f0 con 0x7f397c13fc60 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.940+0000 7f3982ae4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c06fac0 con 0x7f397c0727a0 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.940+0000 7f3982ae4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c0715d0 con 0x7f397c070570 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.940+0000 7f3982ae4640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:28.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.940+0000 7f3982ae4640 10 monclient(hunting): authenticate will time out at 5495.251868s 2024-03-25T19:51:28.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.951+0000 7f3980880640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f397c13fc60 0x7f397c06f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:57506/0 (socket says 172.21.15.71:57506) 2024-03-25T19:51:28.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.951+0000 7f3980880640 1 -- 172.21.15.71:0/1572859881 learned_addr learned my addr 172.21.15.71:0/1572859881 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:28.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178522065 0 0) 0x7f397c0715d0 con 0x7f397c070570 2024-03-25T19:51:28.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:28.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:28.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:28.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 10 monclient(hunting): my global_id is 14708 2024-03-25T19:51:28.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f396c003520 con 0x7f397c070570 2024-03-25T19:51:28.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639921817 0 0) 0x7f397c06fac0 con 0x7f397c0727a0 2024-03-25T19:51:28.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:28.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:28.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:28.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:51:28.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.976+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c0715d0 con 0x7f397c0727a0 2024-03-25T19:51:28.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.983+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717108812 0 0) 0x7f397c13c9f0 con 0x7f397c13fc60 2024-03-25T19:51:28.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.983+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:28.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.983+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:28.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.983+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:28.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.983+0000 7f397b7fe640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:51:28.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:28.983+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c06fac0 con 0x7f397c13fc60 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1621774487 0 0) 0x7f396c003520 con 0x7f397c070570 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 >> v1:172.21.15.193:6789/0 conn(0x7f397c0727a0 legacy=0x7f397c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 >> v1:172.21.15.71:6789/0 conn(0x7f397c13fc60 legacy=0x7f397c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c0718b0 con 0x7f397c070570 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:29.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3968001690 con 0x7f397c070570 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f397b7fe640 10 monclient: dump: 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:29.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f3982ae4640 5 monclient: authenticate success, global_id 14708 2024-03-25T19:51:29.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.001+0000 7f3982ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:29.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.022+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39680017f0 con 0x7f397c070570 2024-03-25T19:51:29.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.022+0000 7f397b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:29.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.022+0000 7f397b7fe640 1 -- 172.21.15.71:0/1572859881 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3968001c00 con 0x7f397c070570 2024-03-25T19:51:29.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.022+0000 7f397b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:29.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.022+0000 7f397b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:29.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.022+0000 7f397b7fe640 10 monclient: dump: 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.022+0000 7f3982ae4640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:29.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.022+0000 7f3982ae4640 4 set_mon_vals no callback set 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.023+0000 7f3982ae4640 10 monclient: shutdown 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.023+0000 7f3982ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.023+0000 7f3982ae4640 1 -- 172.21.15.71:0/1572859881 >> v1:172.21.15.71:6790/0 conn(0x7f397c070570 legacy=0x7f397c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.023+0000 7f3982ae4640 1 -- 172.21.15.71:0/1572859881 shutdown_connections 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.023+0000 7f3982ae4640 1 -- 172.21.15.71:0/1572859881 >> 172.21.15.71:0/1572859881 conn(0x7f397c0b4a80 msgr2=0x7f397c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.023+0000 7f3982ae4640 1 -- 172.21.15.71:0/1572859881 shutdown_connections 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.023+0000 7f3982ae4640 1 -- 172.21.15.71:0/1572859881 wait complete. 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.024+0000 7f3982ae4640 10 monclient: build_initial_monmap 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.024+0000 7f3982ae4640 10 monclient: monmap: 2024-03-25T19:51:29.025 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.025+0000 7f3982ae4640 1 librados: starting msgr at 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.025+0000 7f3982ae4640 1 librados: starting objecter 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.025+0000 7f3982ae4640 1 Processor -- start 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.025+0000 7f3982ae4640 1 -- start start 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.025+0000 7f3982ae4640 1 librados: setting wanted keys 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.025+0000 7f3982ae4640 1 librados: calling monclient init 2024-03-25T19:51:29.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.025+0000 7f3982ae4640 10 monclient: init 2024-03-25T19:51:29.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:29.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 10 monclient(hunting): picked mon.noname-a con 0x7f397c070570 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 10 monclient(hunting): picked mon.noname-c con 0x7f397c0727a0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 10 monclient(hunting): picked mon.noname-b con 0x7f397c13fc60 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c0aaad0 con 0x7f397c070570 2024-03-25T19:51:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c136570 con 0x7f397c13fc60 2024-03-25T19:51:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c1359b0 con 0x7f397c0727a0 2024-03-25T19:51:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.026+0000 7f3982ae4640 10 monclient(hunting): authenticate will time out at 5495.338353s 2024-03-25T19:51:29.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.030+0000 7f3980880640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f397c070570 0x7f397c1d7cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:57554/0 (socket says 172.21.15.71:57554) 2024-03-25T19:51:29.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.030+0000 7f3980880640 1 -- 172.21.15.71:0/2851380337 learned_addr learned my addr 172.21.15.71:0/2851380337 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:29.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.032+0000 7f39797fa640 10 client.?.objecter ms_handle_connect 0x7f397c070570 2024-03-25T19:51:29.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.032+0000 7f39797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:29.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.042+0000 7f39797fa640 10 client.?.objecter ms_handle_connect 0x7f397c13fc60 2024-03-25T19:51:29.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.042+0000 7f39797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:29.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.045+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636550562 0 0) 0x7f397c0aaad0 con 0x7f397c070570 2024-03-25T19:51:29.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.045+0000 7f39797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:29.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.045+0000 7f39797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:29.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.045+0000 7f39797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:29.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.045+0000 7f39797fa640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:51:29.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.045+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3960004eb0 con 0x7f397c070570 2024-03-25T19:51:29.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.051+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369253303 0 0) 0x7f397c136570 con 0x7f397c13fc60 2024-03-25T19:51:29.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.051+0000 7f39797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:29.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.051+0000 7f39797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:29.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.051+0000 7f39797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:29.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.051+0000 7f39797fa640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:51:29.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.051+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c0aaad0 con 0x7f397c13fc60 2024-03-25T19:51:29.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.051+0000 7f39797fa640 10 client.?.objecter ms_handle_connect 0x7f397c0727a0 2024-03-25T19:51:29.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.051+0000 7f39797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:29.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.071+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1367021047 0 0) 0x7f397c0aaad0 con 0x7f397c13fc60 2024-03-25T19:51:29.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.071+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f397c136570 con 0x7f397c13fc60 2024-03-25T19:51:29.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.071+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3974001c70 con 0x7f397c13fc60 2024-03-25T19:51:29.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.071+0000 7f39797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:29.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.071+0000 7f39797fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:29.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.071+0000 7f39797fa640 10 monclient(hunting): dump: 2024-03-25T19:51:29.071 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:29.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.071+0000 7f39797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.073+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1655642305 0 0) 0x7f3960004eb0 con 0x7f397c070570 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.073+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f397c0aaad0 con 0x7f397c070570 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.073+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3970001910 con 0x7f397c070570 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.073+0000 7f39797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.073+0000 7f39797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.073+0000 7f39797fa640 10 monclient(hunting): dump: 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.073 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:29.074 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:29.074 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:29.074 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:29.074 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:29.074 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:29.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.074+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349642892 0 0) 0x7f397c1359b0 con 0x7f397c0727a0 2024-03-25T19:51:29.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.074+0000 7f39797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:29.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.074+0000 7f39797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:29.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.074+0000 7f39797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:29.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.074+0000 7f39797fa640 10 monclient(hunting): my global_id is 14711 2024-03-25T19:51:29.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.074+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3960004eb0 con 0x7f397c0727a0 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2159052232 0 0) 0x7f397c136570 con 0x7f397c13fc60 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 >> v1:172.21.15.71:6790/0 conn(0x7f397c0727a0 legacy=0x7f397c1d8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 >> v1:172.21.15.71:6789/0 conn(0x7f397c070570 legacy=0x7f397c1d7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 1 monclient: found mon.b 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c1d1010 con 0x7f397c13fc60 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:29.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f39797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.078+0000 7f3982ae4640 5 monclient: authenticate success, global_id 4720 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.079+0000 7f3982ae4640 10 monclient: _renew_subs 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.079+0000 7f3982ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.079+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c0a9470 con 0x7f397c13fc60 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.079+0000 7f3982ae4640 10 client.4720.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.079+0000 7f3982ae4640 10 monclient: _renew_subs 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.079+0000 7f3982ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.079+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c0a9c30 con 0x7f397c13fc60 2024-03-25T19:51:29.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.079+0000 7f3982ae4640 1 librados: init done 2024-03-25T19:51:29.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.080+0000 7f3982ae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:29.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.080+0000 7f3982ae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:29.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.080+0000 7f3982ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:29.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.080+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c0627b0 con 0x7f397c13fc60 2024-03-25T19:51:29.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.369+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3974003990 con 0x7f397c13fc60 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.369+0000 7f39797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.369+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f3974003ca0 con 0x7f397c13fc60 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.369+0000 7f39797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.369+0000 7f39797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.369+0000 7f39797fa640 10 monclient: dump: 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:29.369 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:29.370 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:29.370 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:29.370 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:29.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.369+0000 7f397affd640 4 set_mon_vals no callback set 2024-03-25T19:51:29.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.370+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f3974019c70 con 0x7f397c13fc60 2024-03-25T19:51:29.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.370+0000 7f3980880640 10 monclient: get_auth_request con 0x7f396006deb0 auth_method 0 2024-03-25T19:51:29.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.370+0000 7f39797fa640 10 client.4720.objecter ms_handle_connect 0x7f396006deb0 2024-03-25T19:51:29.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.380+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7f3974053910 con 0x7f397c13fc60 2024-03-25T19:51:29.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.380+0000 7f39797fa640 10 client.4720.objecter ms_dispatch 0x7f397c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:29.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.380+0000 7f39797fa640 3 client.4720.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:29.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.380+0000 7f39797fa640 3 client.4720.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:29.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.380+0000 7f39797fa640 20 client.4720.objecter dump_active .. 0 homeless 2024-03-25T19:51:29.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.380+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f397404f420 con 0x7f397c13fc60 2024-03-25T19:51:29.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.380+0000 7f39797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:29.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.380+0000 7f39797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:29.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.585+0000 7f3982ae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:29.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.585+0000 7f3982ae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:29.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.585+0000 7f3982ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:29.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.585+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f397c1d11a0 con 0x7f397c13fc60 2024-03-25T19:51:29.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.647+0000 7f39797fa640 1 -- 172.21.15.71:0/2851380337 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7f3974051350 con 0x7f397c13fc60 2024-03-25T19:51:29.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.647+0000 7f39797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:29.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.647+0000 7f39797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:29.647 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:51:29.647 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.649+0000 7f3982ae4640 10 librados: watch_flush enter 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.649+0000 7f3982ae4640 10 librados: watch_flush exit 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.649+0000 7f3982ae4640 20 client.4720.objecter shutdown clearing up homeless session... 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.649+0000 7f3982ae4640 10 client.4720.objecter successfully canceled tick 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.649+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 >> v1:172.21.15.71:6800/3911735624 conn(0x7f396006deb0 legacy=0x7f3960070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.649+0000 7f3982ae4640 10 monclient: shutdown 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.649+0000 7f3982ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.649+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 >> v1:172.21.15.193:6789/0 conn(0x7f397c13fc60 legacy=0x7f397c1defc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.650+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 shutdown_connections 2024-03-25T19:51:29.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.650+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 >> 172.21.15.71:0/2851380337 conn(0x7f397c0b4a80 msgr2=0x7f397c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:29.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.650+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 shutdown_connections 2024-03-25T19:51:29.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.650+0000 7f3982ae4640 1 -- 172.21.15.71:0/2851380337 wait complete. 2024-03-25T19:51:29.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:29.650+0000 7f3982ae4640 1 librados: shutdown 2024-03-25T19:51:29.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:36.468 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.582+0000 7fcf0eb8e640 10 monclient: get_monmap_and_config 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.582+0000 7fcf0eb8e640 10 monclient: build_initial_monmap 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.582+0000 7fcf0eb8e640 10 monclient: monmap: 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:51:36.584483+0000 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:51:36.584483+0000 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:36.583 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:36.584 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:36.584 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:51:36.584 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:51:36.584 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:36.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.583+0000 7fcf0eb8e640 1 Processor -- start 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.583+0000 7fcf0eb8e640 1 -- start start 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.583+0000 7fcf0eb8e640 10 monclient: init 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 10 monclient(hunting): picked mon.noname-b con 0x7fcf080a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 10 monclient(hunting): picked mon.noname-a con 0x7fcf0813f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 10 monclient(hunting): picked mon.noname-c con 0x7fcf081430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf08142520 con 0x7fcf0813f710 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf080b28a0 con 0x7fcf081430c0 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf080b2a30 con 0x7fcf080a98f0 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:36.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.584+0000 7fcf0eb8e640 10 monclient(hunting): authenticate will time out at 5502.897142s 2024-03-25T19:51:36.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.592+0000 7fcf07fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcf0813f710 0x7fcf081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46356/0 (socket says 172.21.15.71:46356) 2024-03-25T19:51:36.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.592+0000 7fcf07fff640 1 -- 172.21.15.71:0/2067481412 learned_addr learned my addr 172.21.15.71:0/2067481412 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:36.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.615+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346925487 0 0) 0x7fcf08142520 con 0x7fcf0813f710 2024-03-25T19:51:36.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.615+0000 7fcf077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:36.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.615+0000 7fcf077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:36.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.615+0000 7fcf077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:36.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.615+0000 7fcf077fe640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:51:36.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.615+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef0003520 con 0x7fcf0813f710 2024-03-25T19:51:36.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.622+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857814965 0 0) 0x7fcf080b2a30 con 0x7fcf080a98f0 2024-03-25T19:51:36.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.622+0000 7fcf077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:36.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.622+0000 7fcf077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:36.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.622+0000 7fcf077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:36.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.622+0000 7fcf077fe640 10 monclient(hunting): my global_id is 14714 2024-03-25T19:51:36.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.622+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf08142520 con 0x7fcf080a98f0 2024-03-25T19:51:36.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2092683161 0 0) 0x7fcef0003520 con 0x7fcf0813f710 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 >> v1:172.21.15.71:6790/0 conn(0x7fcf080a98f0 legacy=0x7fcf080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 >> v1:172.21.15.193:6789/0 conn(0x7fcf081430c0 legacy=0x7fcf0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 1 monclient: found mon.noname-a 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf080b44c0 con 0x7fcf0813f710 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fcefc001690 con 0x7fcf0813f710 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf077fe640 10 monclient: dump: 2024-03-25T19:51:36.632 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf0eb8e640 5 monclient: authenticate success, global_id 4725 2024-03-25T19:51:36.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.630+0000 7fcf0eb8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:36.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.696+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcefc0017f0 con 0x7fcf0813f710 2024-03-25T19:51:36.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.696+0000 7fcf077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:36.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.696+0000 7fcf077fe640 1 -- 172.21.15.71:0/2067481412 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fcefc001c00 con 0x7fcf0813f710 2024-03-25T19:51:36.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.696+0000 7fcf0eb8e640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:36.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.696+0000 7fcf0eb8e640 4 set_mon_vals no callback set 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.696+0000 7fcf077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.696+0000 7fcf077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.696+0000 7fcf077fe640 10 monclient: dump: 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:36.698 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.697+0000 7fcf0eb8e640 10 monclient: shutdown 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.697+0000 7fcf0eb8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.697+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/2067481412 >> v1:172.21.15.71:6789/0 conn(0x7fcf0813f710 legacy=0x7fcf081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.697+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/2067481412 shutdown_connections 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.697+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/2067481412 >> 172.21.15.71:0/2067481412 conn(0x7fcf080b7ea0 msgr2=0x7fcf080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.697+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/2067481412 shutdown_connections 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.697+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/2067481412 wait complete. 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.698+0000 7fcf0eb8e640 10 monclient: build_initial_monmap 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.698+0000 7fcf0eb8e640 10 monclient: monmap: 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:36.700 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.699+0000 7fcf0eb8e640 1 librados: starting msgr at 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.699+0000 7fcf0eb8e640 1 librados: starting objecter 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.699+0000 7fcf0eb8e640 1 Processor -- start 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.699+0000 7fcf0eb8e640 1 -- start start 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.699+0000 7fcf0eb8e640 1 librados: setting wanted keys 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.699+0000 7fcf0eb8e640 1 librados: calling monclient init 2024-03-25T19:51:36.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.699+0000 7fcf0eb8e640 10 monclient: init 2024-03-25T19:51:36.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 10 monclient(hunting): picked mon.noname-b con 0x7fcf080a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 10 monclient(hunting): picked mon.noname-c con 0x7fcf0813f710 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 10 monclient(hunting): picked mon.noname-a con 0x7fcf081430c0 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf081d82d0 con 0x7fcf081430c0 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf080abb90 con 0x7fcf080a98f0 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf081d76a0 con 0x7fcf0813f710 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:36.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.700+0000 7fcf0eb8e640 10 monclient(hunting): authenticate will time out at 5503.013199s 2024-03-25T19:51:36.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.719+0000 7fcf0d12b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcf081430c0 0x7fcf08134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46390/0 (socket says 172.21.15.71:46390) 2024-03-25T19:51:36.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.719+0000 7fcf0d12b640 1 -- 172.21.15.71:0/241425283 learned_addr learned my addr 172.21.15.71:0/241425283 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:36.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.720+0000 7fcf057fa640 10 client.?.objecter ms_handle_connect 0x7fcf081430c0 2024-03-25T19:51:36.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.720+0000 7fcf057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:36.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.741+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576591674 0 0) 0x7fcf081d82d0 con 0x7fcf081430c0 2024-03-25T19:51:36.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.741+0000 7fcf057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:36.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.741+0000 7fcf057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:36.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.741+0000 7fcf057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:36.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.741+0000 7fcf057fa640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:51:36.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.741+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef4004eb0 con 0x7fcf081430c0 2024-03-25T19:51:36.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.766+0000 7fcf057fa640 10 client.?.objecter ms_handle_connect 0x7fcf080a98f0 2024-03-25T19:51:36.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.766+0000 7fcf057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.768+0000 7fcf057fa640 10 client.?.objecter ms_handle_connect 0x7fcf0813f710 2024-03-25T19:51:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.768+0000 7fcf057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.771+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2532949362 0 0) 0x7fcef4004eb0 con 0x7fcf081430c0 2024-03-25T19:51:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.772+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf081d82d0 con 0x7fcf081430c0 2024-03-25T19:51:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.772+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fcf000018b0 con 0x7fcf081430c0 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.772+0000 7fcf057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.772+0000 7fcf057fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.772+0000 7fcf057fa640 10 monclient(hunting): dump: 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:36.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.772+0000 7fcf057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:36.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.787+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228783711 0 0) 0x7fcf080abb90 con 0x7fcf080a98f0 2024-03-25T19:51:36.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.787+0000 7fcf057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:36.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.787+0000 7fcf057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:36.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.787+0000 7fcf057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:36.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.787+0000 7fcf057fa640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:51:36.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.787+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef4004eb0 con 0x7fcf080a98f0 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176372673 0 0) 0x7fcf081d76a0 con 0x7fcf0813f710 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 10 monclient(hunting): my global_id is 14717 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf080abb90 con 0x7fcf0813f710 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2964896273 0 0) 0x7fcf081d82d0 con 0x7fcf081430c0 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 >> v1:172.21.15.71:6790/0 conn(0x7fcf0813f710 legacy=0x7fcf081d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 >> v1:172.21.15.193:6789/0 conn(0x7fcf080a98f0 legacy=0x7fcf081385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 1 monclient: found mon.a 2024-03-25T19:51:36.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf08134720 con 0x7fcf081430c0 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 5 monclient: authenticate success, global_id 4728 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 10 monclient: _renew_subs 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf08134970 con 0x7fcf081430c0 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 10 client.4728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 10 monclient: _renew_subs 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf08137a10 con 0x7fcf081430c0 2024-03-25T19:51:36.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.789+0000 7fcf0eb8e640 1 librados: init done 2024-03-25T19:51:36.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.791+0000 7fcf0eb8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:36.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.791+0000 7fcf0eb8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:36.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.791+0000 7fcf0eb8e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:36.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:36.791+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf081d7830 con 0x7fcf081430c0 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.131+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf000035f0 con 0x7fcf081430c0 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.131+0000 7fcf057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.131+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7fcf000039c0 con 0x7fcf081430c0 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.131+0000 7fcf057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.131+0000 7fcf057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.131+0000 7fcf057fa640 10 monclient: dump: 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:37.132 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:37.133 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:37.133 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:37.133 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:37.133 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:37.133 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:37.133 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:37.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.131+0000 7fcf06ffd640 4 set_mon_vals no callback set 2024-03-25T19:51:37.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.132+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7fcf00003be0 con 0x7fcf081430c0 2024-03-25T19:51:37.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.133+0000 7fcf0c92a640 10 monclient: get_auth_request con 0x7fcef406de60 auth_method 0 2024-03-25T19:51:37.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.133+0000 7fcf057fa640 10 client.4728.objecter ms_handle_connect 0x7fcef406de60 2024-03-25T19:51:37.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.138+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7fcf000546f0 con 0x7fcf081430c0 2024-03-25T19:51:37.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.138+0000 7fcf057fa640 10 client.4728.objecter ms_dispatch 0x7fcf08000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:37.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.138+0000 7fcf057fa640 3 client.4728.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:37.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.138+0000 7fcf057fa640 3 client.4728.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:37.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.138+0000 7fcf057fa640 20 client.4728.objecter dump_active .. 0 homeless 2024-03-25T19:51:37.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.138+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf00050200 con 0x7fcf081430c0 2024-03-25T19:51:37.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.138+0000 7fcf057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:37.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.138+0000 7fcf057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:37.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.381+0000 7fcf0eb8e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:37.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.381+0000 7fcf0eb8e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:37.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.381+0000 7fcf0eb8e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:37.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.381+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf081d7490 con 0x7fcf081430c0 2024-03-25T19:51:37.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.425+0000 7fcf057fa640 1 -- 172.21.15.71:0/241425283 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7fcf00052130 con 0x7fcf081430c0 2024-03-25T19:51:37.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.425+0000 7fcf057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:37.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.425+0000 7fcf057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:37.426 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:51:37.427 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 10 librados: watch_flush enter 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 10 librados: watch_flush exit 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 20 client.4728.objecter shutdown clearing up homeless session... 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 10 client.4728.objecter successfully canceled tick 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 >> v1:172.21.15.71:6800/3911735624 conn(0x7fcef406de60 legacy=0x7fcef4070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 10 monclient: shutdown 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 >> v1:172.21.15.71:6789/0 conn(0x7fcf081430c0 legacy=0x7fcf08134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 shutdown_connections 2024-03-25T19:51:37.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 >> 172.21.15.71:0/241425283 conn(0x7fcf080b7ea0 msgr2=0x7fcf08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:37.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 shutdown_connections 2024-03-25T19:51:37.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 1 -- 172.21.15.71:0/241425283 wait complete. 2024-03-25T19:51:37.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:37.428+0000 7fcf0eb8e640 1 librados: shutdown 2024-03-25T19:51:37.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:44.247 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:44.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.363+0000 7f9161310640 10 monclient: get_monmap_and_config 2024-03-25T19:51:44.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.363+0000 7f9161310640 10 monclient: build_initial_monmap 2024-03-25T19:51:44.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.363+0000 7f9161310640 10 monclient: monmap: 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:51:44.364723+0000 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:51:44.364723+0000 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:51:44.364 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:44.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.364+0000 7f9161310640 1 Processor -- start 2024-03-25T19:51:44.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.364+0000 7f9161310640 1 -- start start 2024-03-25T19:51:44.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.364+0000 7f9161310640 10 monclient: init 2024-03-25T19:51:44.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:44.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:44.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 10 monclient(hunting): picked mon.noname-b con 0x7f915c0a98f0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:44.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 10 monclient(hunting): picked mon.noname-a con 0x7f915c13f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:44.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 10 monclient(hunting): picked mon.noname-c con 0x7f915c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:44.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c142520 con 0x7f915c13f710 2024-03-25T19:51:44.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c0b28a0 con 0x7f915c1430c0 2024-03-25T19:51:44.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c0b2a30 con 0x7f915c0a98f0 2024-03-25T19:51:44.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:44.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.365+0000 7f9161310640 10 monclient(hunting): authenticate will time out at 5510.677577s 2024-03-25T19:51:44.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.374+0000 7f915affd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f915c0a98f0 0x7f915c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33754/0 (socket says 172.21.15.71:33754) 2024-03-25T19:51:44.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.374+0000 7f915affd640 1 -- 172.21.15.71:0/3652593154 learned_addr learned my addr 172.21.15.71:0/3652593154 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:44.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.393+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250183978 0 0) 0x7f915c142520 con 0x7f915c13f710 2024-03-25T19:51:44.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.393+0000 7f9159ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:44.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.393+0000 7f9159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:44.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.393+0000 7f9159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:44.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.393+0000 7f9159ffb640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:51:44.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.393+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9144003520 con 0x7f915c13f710 2024-03-25T19:51:44.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.397+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343310925 0 0) 0x7f915c0b2a30 con 0x7f915c0a98f0 2024-03-25T19:51:44.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.397+0000 7f9159ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:44.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.397+0000 7f9159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:44.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.397+0000 7f9159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:44.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.397+0000 7f9159ffb640 10 monclient(hunting): my global_id is 14720 2024-03-25T19:51:44.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.397+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915c142520 con 0x7f915c0a98f0 2024-03-25T19:51:44.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.399+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497272454 0 0) 0x7f915c0b28a0 con 0x7f915c1430c0 2024-03-25T19:51:44.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.399+0000 7f9159ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:44.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.399+0000 7f9159ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:44.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.399+0000 7f9159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:44.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.399+0000 7f9159ffb640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:51:44.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.399+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915c0b2a30 con 0x7f915c1430c0 2024-03-25T19:51:44.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3201288988 0 0) 0x7f9144003520 con 0x7f915c13f710 2024-03-25T19:51:44.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 >> v1:172.21.15.71:6790/0 conn(0x7f915c0a98f0 legacy=0x7f915c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:44.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 >> v1:172.21.15.193:6789/0 conn(0x7f915c1430c0 legacy=0x7f915c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:44.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 1 monclient: found mon.noname-a 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c0b44c0 con 0x7f915c13f710 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9150001690 con 0x7f915c13f710 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9159ffb640 10 monclient: dump: 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:44.406 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9161310640 5 monclient: authenticate success, global_id 4731 2024-03-25T19:51:44.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.405+0000 7f9161310640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:44.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.429+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91500017f0 con 0x7f915c13f710 2024-03-25T19:51:44.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.429+0000 7f9159ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:44.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.429+0000 7f9159ffb640 1 -- 172.21.15.71:0/3652593154 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9150001c00 con 0x7f915c13f710 2024-03-25T19:51:44.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.429+0000 7f9161310640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:44.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.429+0000 7f9161310640 4 set_mon_vals no callback set 2024-03-25T19:51:44.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.429+0000 7f9159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:44.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.429+0000 7f9159ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.429+0000 7f9159ffb640 10 monclient: dump: 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:44.430 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.430+0000 7f9161310640 10 monclient: shutdown 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.430+0000 7f9161310640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.430+0000 7f9161310640 1 -- 172.21.15.71:0/3652593154 >> v1:172.21.15.71:6789/0 conn(0x7f915c13f710 legacy=0x7f915c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.430+0000 7f9161310640 1 -- 172.21.15.71:0/3652593154 shutdown_connections 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.430+0000 7f9161310640 1 -- 172.21.15.71:0/3652593154 >> 172.21.15.71:0/3652593154 conn(0x7f915c0b7ea0 msgr2=0x7f915c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.430+0000 7f9161310640 1 -- 172.21.15.71:0/3652593154 shutdown_connections 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.430+0000 7f9161310640 1 -- 172.21.15.71:0/3652593154 wait complete. 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.431+0000 7f9161310640 10 monclient: build_initial_monmap 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.431+0000 7f9161310640 10 monclient: monmap: 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:44.432 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.431+0000 7f9161310640 1 librados: starting msgr at 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.431+0000 7f9161310640 1 librados: starting objecter 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.432+0000 7f9161310640 1 Processor -- start 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.432+0000 7f9161310640 1 -- start start 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.432+0000 7f9161310640 1 librados: setting wanted keys 2024-03-25T19:51:44.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.432+0000 7f9161310640 1 librados: calling monclient init 2024-03-25T19:51:44.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.432+0000 7f9161310640 10 monclient: init 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 10 monclient(hunting): picked mon.noname-b con 0x7f915c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 10 monclient(hunting): picked mon.noname-a con 0x7f915c13f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 10 monclient(hunting): picked mon.noname-c con 0x7f915c1430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c137080 con 0x7f915c13f710 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c137360 con 0x7f915c0a98f0 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c135b10 con 0x7f915c1430c0 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:44.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.433+0000 7f9161310640 10 monclient(hunting): authenticate will time out at 5510.745194s 2024-03-25T19:51:44.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.451+0000 7f915affd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f915c0a98f0 0x7f915c137a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:56922/0 (socket says 172.21.15.71:56922) 2024-03-25T19:51:44.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.451+0000 7f915affd640 1 -- 172.21.15.71:0/1950513781 learned_addr learned my addr 172.21.15.71:0/1950513781 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:44.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.454+0000 7f91437fe640 10 client.?.objecter ms_handle_connect 0x7f915c0a98f0 2024-03-25T19:51:44.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.454+0000 7f91437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:44.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.456+0000 7f91437fe640 10 client.?.objecter ms_handle_connect 0x7f915c1430c0 2024-03-25T19:51:44.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.456+0000 7f91437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:44.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.461+0000 7f91437fe640 10 client.?.objecter ms_handle_connect 0x7f915c13f710 2024-03-25T19:51:44.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.461+0000 7f91437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:44.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.468+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310170930 0 0) 0x7f915c137360 con 0x7f915c0a98f0 2024-03-25T19:51:44.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.468+0000 7f91437fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:44.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.468+0000 7f91437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:44.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.468+0000 7f91437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:44.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.468+0000 7f91437fe640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:51:44.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.468+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9148004eb0 con 0x7f915c0a98f0 2024-03-25T19:51:44.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.471+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642776468 0 0) 0x7f915c137080 con 0x7f915c13f710 2024-03-25T19:51:44.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.471+0000 7f91437fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:44.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.471+0000 7f91437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:44.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.471+0000 7f91437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:44.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.471+0000 7f91437fe640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:51:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.471+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915c137360 con 0x7f915c13f710 2024-03-25T19:51:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.472+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159079385 0 0) 0x7f915c135b10 con 0x7f915c1430c0 2024-03-25T19:51:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.472+0000 7f91437fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.472+0000 7f91437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.472+0000 7f91437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.472+0000 7f91437fe640 10 monclient(hunting): my global_id is 14723 2024-03-25T19:51:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.472+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915c137080 con 0x7f915c1430c0 2024-03-25T19:51:44.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.492+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4091623635 0 0) 0x7f915c137080 con 0x7f915c1430c0 2024-03-25T19:51:44.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.492+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f915c135b10 con 0x7f915c1430c0 2024-03-25T19:51:44.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.492+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9154001bd0 con 0x7f915c1430c0 2024-03-25T19:51:44.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.492+0000 7f91437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:44.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.492+0000 7f91437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:44.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.492+0000 7f91437fe640 10 monclient(hunting): dump: 2024-03-25T19:51:44.492 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:44.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.492+0000 7f91437fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:44.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.494+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2490517368 0 0) 0x7f9148004eb0 con 0x7f915c0a98f0 2024-03-25T19:51:44.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.495+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f915c137080 con 0x7f915c0a98f0 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.495+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f914c0019f0 con 0x7f915c0a98f0 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.495+0000 7f91437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.495+0000 7f91437fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.495+0000 7f91437fe640 10 monclient(hunting): dump: 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:44.495 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.505+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2353421510 0 0) 0x7f915c137360 con 0x7f915c13f710 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.505+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9148004eb0 con 0x7f915c13f710 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.505+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f9150001b00 con 0x7f915c13f710 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.505+0000 7f91437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.505+0000 7f91437fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.505+0000 7f91437fe640 10 monclient(hunting): dump: 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:44.505 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:44.506 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:44.506 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:44.506 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:44.506 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:44.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2663554824 0 0) 0x7f915c137080 con 0x7f915c0a98f0 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 >> v1:172.21.15.71:6790/0 conn(0x7f915c1430c0 legacy=0x7f915c130340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 >> v1:172.21.15.71:6789/0 conn(0x7f915c13f710 legacy=0x7f915c0b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 1 monclient: found mon.b 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c135df0 con 0x7f915c0a98f0 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f91437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 5 monclient: authenticate success, global_id 4729 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 10 monclient: _renew_subs 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915c136340 con 0x7f915c0a98f0 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:44.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 10 monclient: _renew_subs 2024-03-25T19:51:44.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:44.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f915c1352e0 con 0x7f915c0a98f0 2024-03-25T19:51:44.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.508+0000 7f9161310640 1 librados: init done 2024-03-25T19:51:44.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.509+0000 7f9161310640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:44.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.510+0000 7f9161310640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:44.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.510+0000 7f9161310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:44.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.510+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915c074cc0 con 0x7f915c0a98f0 2024-03-25T19:51:44.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.818+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f914c0047e0 con 0x7f915c0a98f0 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.818+0000 7f91437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.818+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f914c004af0 con 0x7f915c0a98f0 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.818+0000 7f91437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.818+0000 7f91437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.818+0000 7f91437fe640 10 monclient: dump: 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:44.818 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:44.819 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:44.819 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:44.819 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:44.819 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:44.819 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:44.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.818+0000 7f91597fa640 4 set_mon_vals no callback set 2024-03-25T19:51:44.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.818+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f914c01ab80 con 0x7f915c0a98f0 2024-03-25T19:51:44.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.819+0000 7f915a7fc640 10 monclient: get_auth_request con 0x7f914806e180 auth_method 0 2024-03-25T19:51:44.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.819+0000 7f91437fe640 10 client.4729.objecter ms_handle_connect 0x7f914806e180 2024-03-25T19:51:44.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.824+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7f914c0548b0 con 0x7f915c0a98f0 2024-03-25T19:51:44.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.824+0000 7f91437fe640 10 client.4729.objecter ms_dispatch 0x7f915c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:44.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.824+0000 7f91437fe640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:44.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.824+0000 7f91437fe640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:44.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.824+0000 7f91437fe640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T19:51:44.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.825+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f914c0501f0 con 0x7f915c0a98f0 2024-03-25T19:51:44.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.825+0000 7f91437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:44.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:44.825+0000 7f91437fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:45.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.074+0000 7f9161310640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:45.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.074+0000 7f9161310640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:45.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.074+0000 7f9161310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:45.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.074+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f915c0627b0 con 0x7f915c0a98f0 2024-03-25T19:51:45.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.217+0000 7f91437fe640 1 -- 172.21.15.71:0/1950513781 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7f914c0522f0 con 0x7f915c0a98f0 2024-03-25T19:51:45.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.217+0000 7f91437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:45.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.217+0000 7f91437fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:45.217 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:51:45.217 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:45.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 10 librados: watch_flush enter 2024-03-25T19:51:45.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 10 librados: watch_flush exit 2024-03-25T19:51:45.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T19:51:45.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 10 client.4729.objecter successfully canceled tick 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 >> v1:172.21.15.71:6800/3911735624 conn(0x7f914806e180 legacy=0x7f9148070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 10 monclient: shutdown 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 >> v1:172.21.15.193:6789/0 conn(0x7f915c0a98f0 legacy=0x7f915c137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 shutdown_connections 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 >> 172.21.15.71:0/1950513781 conn(0x7f915c0b7ea0 msgr2=0x7f915c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 shutdown_connections 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 1 -- 172.21.15.71:0/1950513781 wait complete. 2024-03-25T19:51:45.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:45.220+0000 7f9161310640 1 librados: shutdown 2024-03-25T19:51:45.230 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:48.395 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:31.151 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.467 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:31.223 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.495 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:31.251 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.521 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:31.240 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.528 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:31.247 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.529 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:31.286 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.542 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:31.299 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.557 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:31.276 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.567 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:31.323 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.769 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:31.488 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.770 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:31.489 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.804 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:31.524 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.822 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:31.578 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.826 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:31.582 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.857 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:31.613 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.864 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:31.621 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.865 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:31.621 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:48.908 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:31.664 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:52.037 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.151+0000 7ff68eed6640 10 monclient: get_monmap_and_config 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.151+0000 7ff68eed6640 10 monclient: build_initial_monmap 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.151+0000 7ff68eed6640 10 monclient: monmap: 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:51:52.151927+0000 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:51:52.151927+0000 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:51:52.151 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:52.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.152+0000 7ff68eed6640 1 Processor -- start 2024-03-25T19:51:52.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.152+0000 7ff68eed6640 1 -- start start 2024-03-25T19:51:52.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.152+0000 7ff68eed6640 10 monclient: init 2024-03-25T19:51:52.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.152+0000 7ff68eed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:52.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.152+0000 7ff68eed6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:52.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.153+0000 7ff68eed6640 10 monclient(hunting): picked mon.noname-b con 0x7ff6880b4d60 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:52.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.153+0000 7ff68eed6640 10 monclient(hunting): picked mon.noname-c con 0x7ff6880b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:52.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.153+0000 7ff68eed6640 10 monclient(hunting): picked mon.noname-a con 0x7ff68814be90 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:52.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.153+0000 7ff68eed6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6880af940 con 0x7ff68814be90 2024-03-25T19:51:52.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.153+0000 7ff68eed6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6880b0a70 con 0x7ff6880b1730 2024-03-25T19:51:52.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.153+0000 7ff68eed6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6880b0d50 con 0x7ff6880b4d60 2024-03-25T19:51:52.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.153+0000 7ff68eed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:52.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.153+0000 7ff68eed6640 10 monclient(hunting): authenticate will time out at 5518.464938s 2024-03-25T19:51:52.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.155+0000 7ff68d473640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff68814be90 0x7ff68814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34094/0 (socket says 172.21.15.71:34094) 2024-03-25T19:51:52.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.155+0000 7ff68d473640 1 -- 172.21.15.71:0/2183554852 learned_addr learned my addr 172.21.15.71:0/2183554852 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:52.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.174+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156918471 0 0) 0x7ff6880b0a70 con 0x7ff6880b1730 2024-03-25T19:51:52.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.174+0000 7ff6877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:52.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.174+0000 7ff6877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:52.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.174+0000 7ff6877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:52.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.174+0000 7ff6877fe640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:51:52.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.174+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff678003520 con 0x7ff6880b1730 2024-03-25T19:51:52.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.177+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862760601 0 0) 0x7ff6880af940 con 0x7ff68814be90 2024-03-25T19:51:52.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.177+0000 7ff6877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:52.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.177+0000 7ff6877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:52.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.177+0000 7ff6877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:52.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.177+0000 7ff6877fe640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:51:52.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.177+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6880b0a70 con 0x7ff68814be90 2024-03-25T19:51:52.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.181+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196898179 0 0) 0x7ff6880b0d50 con 0x7ff6880b4d60 2024-03-25T19:51:52.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.181+0000 7ff6877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:52.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.181+0000 7ff6877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:52.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.181+0000 7ff6877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:52.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.181+0000 7ff6877fe640 10 monclient(hunting): my global_id is 14726 2024-03-25T19:51:52.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.181+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6880af940 con 0x7ff6880b4d60 2024-03-25T19:51:52.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1946949142 0 0) 0x7ff6880b0a70 con 0x7ff68814be90 2024-03-25T19:51:52.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 >> v1:172.21.15.71:6790/0 conn(0x7ff6880b4d60 legacy=0x7ff68813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:52.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 >> v1:172.21.15.193:6789/0 conn(0x7ff6880b1730 legacy=0x7ff6880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:52.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:52.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 1 monclient: found mon.noname-a 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6880b1030 con 0x7ff68814be90 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff68eed6640 5 monclient: authenticate success, global_id 4737 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff6800019f0 con 0x7ff68814be90 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff6877fe640 10 monclient: dump: 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:52.187 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:52.188 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:52.188 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:52.188 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:52.188 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:52.188 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:52.188 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:52.188 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:52.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.186+0000 7ff68eed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.199+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff680001b50 con 0x7ff68814be90 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.199+0000 7ff6877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.199+0000 7ff6877fe640 1 -- 172.21.15.71:0/2183554852 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff680001f60 con 0x7ff68814be90 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.199+0000 7ff6877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.199+0000 7ff6877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.199+0000 7ff6877fe640 10 monclient: dump: 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:52.199 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:52.200 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:52.200 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:52.200 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:52.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.199+0000 7ff68eed6640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:52.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.199+0000 7ff68eed6640 4 set_mon_vals no callback set 2024-03-25T19:51:52.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.200+0000 7ff68eed6640 10 monclient: shutdown 2024-03-25T19:51:52.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.200+0000 7ff68eed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:52.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.200+0000 7ff68eed6640 1 -- 172.21.15.71:0/2183554852 >> v1:172.21.15.71:6789/0 conn(0x7ff68814be90 legacy=0x7ff68814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:52.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.200+0000 7ff68eed6640 1 -- 172.21.15.71:0/2183554852 shutdown_connections 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.200+0000 7ff68eed6640 1 -- 172.21.15.71:0/2183554852 >> 172.21.15.71:0/2183554852 conn(0x7ff6880617a0 msgr2=0x7ff688061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.200+0000 7ff68eed6640 1 -- 172.21.15.71:0/2183554852 shutdown_connections 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.200+0000 7ff68eed6640 1 -- 172.21.15.71:0/2183554852 wait complete. 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 10 monclient: build_initial_monmap 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 10 monclient: monmap: 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:51:52.202 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:51:52.203 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:52.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 1 librados: starting msgr at 2024-03-25T19:51:52.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 1 librados: starting objecter 2024-03-25T19:51:52.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 1 Processor -- start 2024-03-25T19:51:52.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 1 -- start start 2024-03-25T19:51:52.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 1 librados: setting wanted keys 2024-03-25T19:51:52.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 1 librados: calling monclient init 2024-03-25T19:51:52.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.201+0000 7ff68eed6640 10 monclient: init 2024-03-25T19:51:52.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.202+0000 7ff68eed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:52.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.202+0000 7ff68eed6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:52.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.202+0000 7ff68eed6640 10 monclient(hunting): picked mon.noname-b con 0x7ff6880b1730 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:52.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.202+0000 7ff68eed6640 10 monclient(hunting): picked mon.noname-a con 0x7ff6880b4d60 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:52.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.202+0000 7ff68eed6640 10 monclient(hunting): picked mon.noname-c con 0x7ff68814be90 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:52.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.202+0000 7ff68eed6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6881327e0 con 0x7ff6880b4d60 2024-03-25T19:51:52.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.203+0000 7ff68eed6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6881408f0 con 0x7ff6880b1730 2024-03-25T19:51:52.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.203+0000 7ff68eed6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff688140df0 con 0x7ff68814be90 2024-03-25T19:51:52.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.203+0000 7ff68eed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:52.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.203+0000 7ff68eed6640 10 monclient(hunting): authenticate will time out at 5518.514863s 2024-03-25T19:51:52.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.208+0000 7ff68cc72640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff6880b1730 0x7ff6881dad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:35474/0 (socket says 172.21.15.71:35474) 2024-03-25T19:51:52.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.208+0000 7ff68cc72640 1 -- 172.21.15.71:0/2071843805 learned_addr learned my addr 172.21.15.71:0/2071843805 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:52.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.209+0000 7ff6857fa640 10 client.?.objecter ms_handle_connect 0x7ff6880b1730 2024-03-25T19:51:52.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.209+0000 7ff6857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:52.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.213+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850322907 0 0) 0x7ff6881408f0 con 0x7ff6880b1730 2024-03-25T19:51:52.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.213+0000 7ff6857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:52.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.213+0000 7ff6857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:52.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.213+0000 7ff6857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:52.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.213+0000 7ff6857fa640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:51:52.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.213+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff66c004eb0 con 0x7ff6880b1730 2024-03-25T19:51:52.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.221+0000 7ff6857fa640 10 client.?.objecter ms_handle_connect 0x7ff6880b4d60 2024-03-25T19:51:52.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.221+0000 7ff6857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:52.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.221+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184313877 0 0) 0x7ff66c004eb0 con 0x7ff6880b1730 2024-03-25T19:51:52.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.221+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6881408f0 con 0x7ff6880b1730 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.221+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff67c001a10 con 0x7ff6880b1730 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.221+0000 7ff6857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.222+0000 7ff6857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.222+0000 7ff6857fa640 10 monclient(hunting): dump: 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:52.222 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:52.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.222+0000 7ff6857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:52.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.226+0000 7ff6857fa640 10 client.?.objecter ms_handle_connect 0x7ff68814be90 2024-03-25T19:51:52.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.226+0000 7ff6857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:52.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1581143172 0 0) 0x7ff6881408f0 con 0x7ff6880b1730 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 >> v1:172.21.15.71:6790/0 conn(0x7ff68814be90 legacy=0x7ff6881320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 >> v1:172.21.15.71:6789/0 conn(0x7ff6880b4d60 legacy=0x7ff6881319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 1 monclient: found mon.b 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff688140380 con 0x7ff6880b1730 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff6857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 5 monclient: authenticate success, global_id 4735 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 10 monclient: _renew_subs 2024-03-25T19:51:52.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:52.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff688140630 con 0x7ff6880b1730 2024-03-25T19:51:52.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:52.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 10 monclient: _renew_subs 2024-03-25T19:51:52.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:52.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6881db620 con 0x7ff6880b1730 2024-03-25T19:51:52.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.231+0000 7ff68eed6640 1 librados: init done 2024-03-25T19:51:52.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.232+0000 7ff68eed6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:52.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.233+0000 7ff68eed6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:52.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.233+0000 7ff68eed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:52.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.233+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68813cc20 con 0x7ff6880b1730 2024-03-25T19:51:52.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.516+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff67c001cf0 con 0x7ff6880b1730 2024-03-25T19:51:52.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.517+0000 7ff6857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.517+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7ff67c003a60 con 0x7ff6880b1730 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.517+0000 7ff6857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.517+0000 7ff6857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.517+0000 7ff6857fa640 10 monclient: dump: 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:52.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.517+0000 7ff686ffd640 4 set_mon_vals no callback set 2024-03-25T19:51:52.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.518+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7ff67c003c80 con 0x7ff6880b1730 2024-03-25T19:51:52.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.518+0000 7ff687fff640 10 monclient: get_auth_request con 0x7ff66c06dd00 auth_method 0 2024-03-25T19:51:52.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.518+0000 7ff6857fa640 10 client.4735.objecter ms_handle_connect 0x7ff66c06dd00 2024-03-25T19:51:52.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.524+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7ff67c054860 con 0x7ff6880b1730 2024-03-25T19:51:52.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.524+0000 7ff6857fa640 10 client.4735.objecter ms_dispatch 0x7ff688000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:52.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.524+0000 7ff6857fa640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:52.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.524+0000 7ff6857fa640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:52.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.524+0000 7ff6857fa640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T19:51:52.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.527+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff67c050370 con 0x7ff6880b1730 2024-03-25T19:51:52.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.527+0000 7ff6857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:52.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.527+0000 7ff6857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:52.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.773+0000 7ff68eed6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:52.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.773+0000 7ff68eed6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:52.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.773+0000 7ff68eed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T19:51:52.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.773+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff688140a80 con 0x7ff6880b1730 2024-03-25T19:51:52.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.803+0000 7ff6857fa640 1 -- 172.21.15.71:0/2071843805 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7ff67c0522a0 con 0x7ff6880b1730 2024-03-25T19:51:52.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.803+0000 7ff6857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:52.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.803+0000 7ff6857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:52.803 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:51:52.803 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.805+0000 7ff68eed6640 10 librados: watch_flush enter 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.805+0000 7ff68eed6640 10 librados: watch_flush exit 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.805+0000 7ff68eed6640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.805+0000 7ff68eed6640 10 client.4735.objecter successfully canceled tick 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 >> v1:172.21.15.71:6800/3911735624 conn(0x7ff66c06dd00 legacy=0x7ff66c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 10 monclient: shutdown 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 >> v1:172.21.15.193:6789/0 conn(0x7ff6880b1730 legacy=0x7ff6881dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 shutdown_connections 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 >> 172.21.15.71:0/2071843805 conn(0x7ff6880617a0 msgr2=0x7ff688061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:52.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 shutdown_connections 2024-03-25T19:51:52.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 1 -- 172.21.15.71:0/2071843805 wait complete. 2024-03-25T19:51:52.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:52.806+0000 7ff68eed6640 1 librados: shutdown 2024-03-25T19:51:52.817 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:59.624 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:59.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.736+0000 7f8dd5a0e640 10 monclient: get_monmap_and_config 2024-03-25T19:51:59.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.736+0000 7f8dd5a0e640 10 monclient: build_initial_monmap 2024-03-25T19:51:59.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.736+0000 7f8dd5a0e640 10 monclient: monmap: 2024-03-25T19:51:59.736 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:59.736 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:59.737 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:51:59.737703+0000 2024-03-25T19:51:59.737 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:51:59.737703+0000 2024-03-25T19:51:59.737 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:59.737 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:59.737 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:59.737 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:51:59.737 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:51:59.737 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:59.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.737+0000 7f8dd5a0e640 1 Processor -- start 2024-03-25T19:51:59.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.737+0000 7f8dd5a0e640 1 -- start start 2024-03-25T19:51:59.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.737+0000 7f8dd5a0e640 10 monclient: init 2024-03-25T19:51:59.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd00705b0 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd01402d0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd0149ae0 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd00718c0 con 0x7f8dd00705b0 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd00714c0 con 0x7f8dd0149ae0 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd0072b50 con 0x7f8dd01402d0 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:59.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.738+0000 7f8dd5a0e640 10 monclient(hunting): authenticate will time out at 5526.050459s 2024-03-25T19:51:59.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.745+0000 7f8dcf7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f8dd0149ae0 0x7f8dd014bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:35564/0 (socket says 172.21.15.71:35564) 2024-03-25T19:51:59.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.745+0000 7f8dcf7fe640 1 -- 172.21.15.71:0/2285794273 learned_addr learned my addr 172.21.15.71:0/2285794273 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:59.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.782+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806991232 0 0) 0x7f8dd00718c0 con 0x7f8dd00705b0 2024-03-25T19:51:59.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.782+0000 7f8dcdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:59.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.782+0000 7f8dcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:59.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.782+0000 7f8dcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:59.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.782+0000 7f8dcdffb640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:51:59.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.782+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db8003520 con 0x7f8dd00705b0 2024-03-25T19:51:59.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581935080 0 0) 0x7f8db8003520 con 0x7f8dd00705b0 2024-03-25T19:51:59.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 >> v1:172.21.15.71:6790/0 conn(0x7f8dd01402d0 legacy=0x7f8dd0140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:59.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 >> v1:172.21.15.193:6789/0 conn(0x7f8dd0149ae0 legacy=0x7f8dd014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 1 monclient: found mon.noname-a 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd013dd10 con 0x7f8dd00705b0 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8dc0001690 con 0x7f8dd00705b0 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dcdffb640 10 monclient: dump: 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:59.847 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dd5a0e640 5 monclient: authenticate success, global_id 4743 2024-03-25T19:51:59.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.846+0000 7f8dd5a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:59.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.861+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dc00017f0 con 0x7f8dd00705b0 2024-03-25T19:51:59.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.861+0000 7f8dcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:59.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.861+0000 7f8dcdffb640 1 -- 172.21.15.71:0/2285794273 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8dc0001c00 con 0x7f8dd00705b0 2024-03-25T19:51:59.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.861+0000 7f8dcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:59.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.861+0000 7f8dcdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:59.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.861+0000 7f8dcdffb640 10 monclient: dump: 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.861+0000 7f8dd5a0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:59.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.861+0000 7f8dd5a0e640 4 set_mon_vals no callback set 2024-03-25T19:51:59.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.862+0000 7f8dd5a0e640 10 monclient: shutdown 2024-03-25T19:51:59.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.862+0000 7f8dd5a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:59.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.862+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/2285794273 >> v1:172.21.15.71:6789/0 conn(0x7f8dd00705b0 legacy=0x7f8dd013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:59.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.862+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/2285794273 shutdown_connections 2024-03-25T19:51:59.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.862+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/2285794273 >> 172.21.15.71:0/2285794273 conn(0x7f8dd00b5d10 msgr2=0x7f8dd00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:59.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.862+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/2285794273 shutdown_connections 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.862+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/2285794273 wait complete. 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.863+0000 7f8dd5a0e640 10 monclient: build_initial_monmap 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.863+0000 7f8dd5a0e640 10 monclient: monmap: 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:59.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.864+0000 7f8dd5a0e640 1 librados: starting msgr at 2024-03-25T19:51:59.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.864+0000 7f8dd5a0e640 1 librados: starting objecter 2024-03-25T19:51:59.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.864+0000 7f8dd5a0e640 1 Processor -- start 2024-03-25T19:51:59.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.864+0000 7f8dd5a0e640 1 -- start start 2024-03-25T19:51:59.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.864+0000 7f8dd5a0e640 1 librados: setting wanted keys 2024-03-25T19:51:59.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.864+0000 7f8dd5a0e640 1 librados: calling monclient init 2024-03-25T19:51:59.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.864+0000 7f8dd5a0e640 10 monclient: init 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd00705b0 addr v1:172.21.15.71:6789/0 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd01402d0 addr v1:172.21.15.71:6790/0 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd0149ae0 addr v1:172.21.15.193:6789/0 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd0144520 con 0x7f8dd00705b0 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd0142ca0 con 0x7f8dd0149ae0 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd0142e30 con 0x7f8dd01402d0 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:59.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.865+0000 7f8dd5a0e640 10 monclient(hunting): authenticate will time out at 5526.177681s 2024-03-25T19:51:59.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.875+0000 7f8dce7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8dd01402d0 0x7f8dd01d9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:59762/0 (socket says 172.21.15.71:59762) 2024-03-25T19:51:59.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.875+0000 7f8dce7fc640 1 -- 172.21.15.71:0/1787757560 learned_addr learned my addr 172.21.15.71:0/1787757560 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:51:59.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.876+0000 7f8db7fff640 10 client.?.objecter ms_handle_connect 0x7f8dd01402d0 2024-03-25T19:51:59.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.876+0000 7f8db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:59.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.876+0000 7f8db7fff640 10 client.?.objecter ms_handle_connect 0x7f8dd00705b0 2024-03-25T19:51:59.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.876+0000 7f8db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:59.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.883+0000 7f8db7fff640 10 client.?.objecter ms_handle_connect 0x7f8dd0149ae0 2024-03-25T19:51:59.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.883+0000 7f8db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:59.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.885+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092008604 0 0) 0x7f8dd0142e30 con 0x7f8dd01402d0 2024-03-25T19:51:59.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.885+0000 7f8db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:59.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.885+0000 7f8db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:59.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.885+0000 7f8db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:59.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.885+0000 7f8db7fff640 10 monclient(hunting): my global_id is 14735 2024-03-25T19:51:59.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.885+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dbc004eb0 con 0x7f8dd01402d0 2024-03-25T19:51:59.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.897+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714584039 0 0) 0x7f8dd0144520 con 0x7f8dd00705b0 2024-03-25T19:51:59.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.897+0000 7f8db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:59.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.897+0000 7f8db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:59.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.897+0000 7f8db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:59.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.897+0000 7f8db7fff640 10 monclient(hunting): my global_id is 4746 2024-03-25T19:51:59.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.897+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd0142e30 con 0x7f8dd00705b0 2024-03-25T19:51:59.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.908+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3741392460 0 0) 0x7f8dd0142e30 con 0x7f8dd00705b0 2024-03-25T19:51:59.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.908+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dd0144520 con 0x7f8dd00705b0 2024-03-25T19:51:59.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.908+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8dc0001b80 con 0x7f8dd00705b0 2024-03-25T19:51:59.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.908+0000 7f8db7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:59.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.908+0000 7f8db7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.908+0000 7f8db7fff640 10 monclient(hunting): dump: 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:59.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.909+0000 7f8db7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:59.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.914+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111635809 0 0) 0x7f8dbc004eb0 con 0x7f8dd01402d0 2024-03-25T19:51:59.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.914+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dd0142e30 con 0x7f8dd01402d0 2024-03-25T19:51:59.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.914+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8dc40019d0 con 0x7f8dd01402d0 2024-03-25T19:51:59.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.914+0000 7f8db7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:59.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.914+0000 7f8db7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:59.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.914+0000 7f8db7fff640 10 monclient(hunting): dump: 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:51:59.915 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:51:59.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.921+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486421786 0 0) 0x7f8dd0142ca0 con 0x7f8dd0149ae0 2024-03-25T19:51:59.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.921+0000 7f8db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:59.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.921+0000 7f8db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:59.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.921+0000 7f8db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:59.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.921+0000 7f8db7fff640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:51:59.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.921+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dbc004eb0 con 0x7f8dd0149ae0 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 313004384 0 0) 0x7f8dd0144520 con 0x7f8dd00705b0 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 >> v1:172.21.15.71:6790/0 conn(0x7f8dd01402d0 legacy=0x7f8dd01d9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 >> v1:172.21.15.193:6789/0 conn(0x7f8dd0149ae0 legacy=0x7f8dd01dfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 1 monclient: found mon.a 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0141800 con 0x7f8dd00705b0 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 10 monclient: _finish_auth 0 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 5 monclient: authenticate success, global_id 4746 2024-03-25T19:51:59.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 10 monclient: _renew_subs 2024-03-25T19:51:59.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:59.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd0141bc0 con 0x7f8dd00705b0 2024-03-25T19:51:59.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 10 client.4746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:59.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 10 monclient: _renew_subs 2024-03-25T19:51:59.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:59.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd0142460 con 0x7f8dd00705b0 2024-03-25T19:51:59.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.930+0000 7f8dd5a0e640 1 librados: init done 2024-03-25T19:51:59.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.931+0000 7f8dd5a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:59.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.931+0000 7f8dd5a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:59.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.931+0000 7f8dd5a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:51:59.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:51:59.931+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd0144340 con 0x7f8dd00705b0 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.237+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dc00038a0 con 0x7f8dd00705b0 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.237+0000 7f8db7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.237+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f8dc0003bb0 con 0x7f8dd00705b0 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.237+0000 7f8db7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.237+0000 7f8db7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.237+0000 7f8db7fff640 10 monclient: dump: 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:00.237 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:52:00.238 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:52:00.238 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:52:00.238 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:52:00.238 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:52:00.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.237+0000 7f8dcd7fa640 4 set_mon_vals no callback set 2024-03-25T19:52:00.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.238+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f8dc0019b80 con 0x7f8dd00705b0 2024-03-25T19:52:00.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.238+0000 7f8dce7fc640 10 monclient: get_auth_request con 0x7f8dbc06df50 auth_method 0 2024-03-25T19:52:00.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.239+0000 7f8db7fff640 10 client.4746.objecter ms_handle_connect 0x7f8dbc06df50 2024-03-25T19:52:00.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.246+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7f8dc0053700 con 0x7f8dd00705b0 2024-03-25T19:52:00.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.246+0000 7f8db7fff640 10 client.4746.objecter ms_dispatch 0x7f8dd0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:00.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.246+0000 7f8db7fff640 3 client.4746.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:00.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.246+0000 7f8db7fff640 3 client.4746.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:00.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.246+0000 7f8db7fff640 20 client.4746.objecter dump_active .. 0 homeless 2024-03-25T19:52:00.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.246+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8dc004f210 con 0x7f8dd00705b0 2024-03-25T19:52:00.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.246+0000 7f8db7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:00.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.246+0000 7f8db7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:00.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.516+0000 7f8dd5a0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:00.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.516+0000 7f8dd5a0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:00.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.516+0000 7f8dd5a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:52:00.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.516+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dd00627b0 con 0x7f8dd00705b0 2024-03-25T19:52:00.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.555+0000 7f8db7fff640 1 -- 172.21.15.71:0/1787757560 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7f8dc0051140 con 0x7f8dd00705b0 2024-03-25T19:52:00.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.555+0000 7f8db7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:00.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.555+0000 7f8db7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:00.556 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:52:00.556 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:00.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 10 librados: watch_flush enter 2024-03-25T19:52:00.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 10 librados: watch_flush exit 2024-03-25T19:52:00.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 20 client.4746.objecter shutdown clearing up homeless session... 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 10 client.4746.objecter successfully canceled tick 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 >> v1:172.21.15.71:6800/3911735624 conn(0x7f8dbc06df50 legacy=0x7f8dbc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 10 monclient: shutdown 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 >> v1:172.21.15.71:6789/0 conn(0x7f8dd00705b0 legacy=0x7f8dd01d8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 shutdown_connections 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.557+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 >> 172.21.15.71:0/1787757560 conn(0x7f8dd00b5d10 msgr2=0x7f8dd00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.558+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 shutdown_connections 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.558+0000 7f8dd5a0e640 1 -- 172.21.15.71:0/1787757560 wait complete. 2024-03-25T19:52:00.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:00.558+0000 7f8dd5a0e640 1 librados: shutdown 2024-03-25T19:52:00.570 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:07.375 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:07.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.493+0000 7f98d596c640 10 monclient: get_monmap_and_config 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.493+0000 7f98d596c640 10 monclient: build_initial_monmap 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.493+0000 7f98d596c640 10 monclient: monmap: 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:52:07.493991+0000 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:52:07.493991+0000 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T19:52:07.524 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-25T19:52:07.525 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.493+0000 7f98d596c640 1 Processor -- start 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.493+0000 7f98d596c640 1 -- start start 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.493+0000 7f98d596c640 10 monclient: init 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 10 monclient(hunting): picked mon.noname-c con 0x7f98d00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 10 monclient(hunting): picked mon.noname-a con 0x7f98d013f710 addr v1:172.21.15.71:6789/0 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 10 monclient(hunting): picked mon.noname-b con 0x7f98d01430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0142520 con 0x7f98d013f710 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00b28a0 con 0x7f98d00a98f0 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00b2a30 con 0x7f98d01430c0 2024-03-25T19:52:07.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.494+0000 7f98d596c640 10 monclient(hunting): authenticate will time out at 5533.806307s 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.499+0000 7f98ceffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f98d00a98f0 0x7f98d00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:40080/0 (socket says 172.21.15.71:40080) 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.499+0000 7f98ceffd640 1 -- 172.21.15.71:0/957753971 learned_addr learned my addr 172.21.15.71:0/957753971 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.504+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538443235 0 0) 0x7f98d00b28a0 con 0x7f98d00a98f0 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.504+0000 7f98cdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.504+0000 7f98cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.504+0000 7f98cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.504+0000 7f98cdffb640 10 monclient(hunting): my global_id is 4744 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.504+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b8003520 con 0x7f98d00a98f0 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2921715538 0 0) 0x7f98b8003520 con 0x7f98d00a98f0 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 >> v1:172.21.15.71:6790/0 conn(0x7f98d01430c0 legacy=0x7f98d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 >> v1:172.21.15.71:6789/0 conn(0x7f98d013f710 legacy=0x7f98d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:52:07.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 1 monclient: found mon.noname-c 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00b44c0 con 0x7f98d00a98f0 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 10 monclient: _finish_auth 0 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f98c0001690 con 0x7f98d00a98f0 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98cdffb640 10 monclient: dump: 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.530 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98d596c640 5 monclient: authenticate success, global_id 4744 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.511+0000 7f98d596c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.517+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c0001860 con 0x7f98d00a98f0 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.517+0000 7f98cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.517+0000 7f98cdffb640 1 -- 172.21.15.71:0/957753971 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f98c0001cb0 con 0x7f98d00a98f0 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.517+0000 7f98cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.517+0000 7f98cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.517+0000 7f98cdffb640 10 monclient: dump: 2024-03-25T19:52:07.531 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.517+0000 7f98d596c640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.517+0000 7f98d596c640 4 set_mon_vals no callback set 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.518+0000 7f98d596c640 10 monclient: shutdown 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.518+0000 7f98d596c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.518+0000 7f98d596c640 1 -- 172.21.15.71:0/957753971 >> v1:172.21.15.193:6789/0 conn(0x7f98d00a98f0 legacy=0x7f98d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.518+0000 7f98d596c640 1 -- 172.21.15.71:0/957753971 shutdown_connections 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.518+0000 7f98d596c640 1 -- 172.21.15.71:0/957753971 >> 172.21.15.71:0/957753971 conn(0x7f98d00b7ea0 msgr2=0x7f98d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:07.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.518+0000 7f98d596c640 1 -- 172.21.15.71:0/957753971 shutdown_connections 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.518+0000 7f98d596c640 1 -- 172.21.15.71:0/957753971 wait complete. 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.519+0000 7f98d596c640 10 monclient: build_initial_monmap 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.519+0000 7f98d596c640 10 monclient: monmap: 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.520+0000 7f98d596c640 1 librados: starting msgr at 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.520+0000 7f98d596c640 1 librados: starting objecter 2024-03-25T19:52:07.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.520+0000 7f98d596c640 1 Processor -- start 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.520+0000 7f98d596c640 1 -- start start 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.520+0000 7f98d596c640 1 librados: setting wanted keys 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.520+0000 7f98d596c640 1 librados: calling monclient init 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.520+0000 7f98d596c640 10 monclient: init 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 10 monclient(hunting): picked mon.noname-a con 0x7f98d00a98f0 addr v1:172.21.15.71:6789/0 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 10 monclient(hunting): picked mon.noname-b con 0x7f98d013f710 addr v1:172.21.15.193:6789/0 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 10 monclient(hunting): picked mon.noname-c con 0x7f98d01430c0 addr v1:172.21.15.71:6790/0 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d013a200 con 0x7f98d00a98f0 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d013b660 con 0x7f98d013f710 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0139d20 con 0x7f98d01430c0 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:07.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.521+0000 7f98d596c640 10 monclient(hunting): authenticate will time out at 5533.833209s 2024-03-25T19:52:07.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.531+0000 7f98ce7fc640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f98d013f710 0x7f98d01dbb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.71:40108/0 (socket says 172.21.15.71:40108) 2024-03-25T19:52:07.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.531+0000 7f98ce7fc640 1 -- 172.21.15.71:0/396576583 learned_addr learned my addr 172.21.15.71:0/396576583 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-25T19:52:07.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.532+0000 7f98b7fff640 10 client.?.objecter ms_handle_connect 0x7f98d013f710 2024-03-25T19:52:07.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.532+0000 7f98b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:07.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.533+0000 7f98b7fff640 10 client.?.objecter ms_handle_connect 0x7f98d00a98f0 2024-03-25T19:52:07.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.533+0000 7f98b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:07.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.538+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298353832 0 0) 0x7f98d013a200 con 0x7f98d00a98f0 2024-03-25T19:52:07.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.538+0000 7f98b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:07.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.538+0000 7f98b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:07.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.538+0000 7f98b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:07.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.538+0000 7f98b7fff640 10 monclient(hunting): my global_id is 4749 2024-03-25T19:52:07.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.538+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc004eb0 con 0x7f98d00a98f0 2024-03-25T19:52:07.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.542+0000 7f98b7fff640 10 client.?.objecter ms_handle_connect 0x7f98d01430c0 2024-03-25T19:52:07.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.542+0000 7f98b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:07.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.544+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690760782 0 0) 0x7f98d013b660 con 0x7f98d013f710 2024-03-25T19:52:07.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.544+0000 7f98b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:07.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.544+0000 7f98b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:07.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.544+0000 7f98b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:07.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.544+0000 7f98b7fff640 10 monclient(hunting): my global_id is 4747 2024-03-25T19:52:07.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.544+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d013a200 con 0x7f98d013f710 2024-03-25T19:52:07.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.551+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535809445 0 0) 0x7f98d0139d20 con 0x7f98d01430c0 2024-03-25T19:52:07.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.551+0000 7f98b7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:07.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.551+0000 7f98b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:07.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.551+0000 7f98b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:07.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.551+0000 7f98b7fff640 10 monclient(hunting): my global_id is 14741 2024-03-25T19:52:07.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.551+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d013b660 con 0x7f98d01430c0 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.554+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 832189743 0 0) 0x7f98bc004eb0 con 0x7f98d00a98f0 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.554+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d0139d20 con 0x7f98d00a98f0 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.554+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f98c0001b70 con 0x7f98d00a98f0 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.554+0000 7f98b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.554+0000 7f98b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.554+0000 7f98b7fff640 10 monclient(hunting): dump: 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:07.554 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:52:07.555 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:52:07.555 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:52:07.555 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:52:07.555 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:52:07.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.554+0000 7f98b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.563+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2486717793 0 0) 0x7f98d013a200 con 0x7f98d013f710 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.563+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98bc004eb0 con 0x7f98d013f710 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.563+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f98c4004000 con 0x7f98d013f710 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.563+0000 7f98b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.563+0000 7f98b7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.563+0000 7f98b7fff640 10 monclient(hunting): dump: 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:52:07.563 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:52:07.564 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:52:07.564 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1249166818 0 0) 0x7f98d0139d20 con 0x7f98d00a98f0 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 >> v1:172.21.15.71:6790/0 conn(0x7f98d01430c0 legacy=0x7f98d01e73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 >> v1:172.21.15.193:6789/0 conn(0x7f98d013f710 legacy=0x7f98d01dbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 1 monclient: found mon.a 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d0139eb0 con 0x7f98d00a98f0 2024-03-25T19:52:07.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 10 monclient: _finish_auth 0 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 5 monclient: authenticate success, global_id 4749 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 10 monclient: _renew_subs 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d0135830 con 0x7f98d00a98f0 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 10 client.4749.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 10 monclient: _renew_subs 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d014d4d0 con 0x7f98d00a98f0 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.564+0000 7f98d596c640 1 librados: init done 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.565+0000 7f98d596c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.565+0000 7f98d596c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:07.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.565+0000 7f98d596c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:52:07.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.565+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d014f2d0 con 0x7f98d00a98f0 2024-03-25T19:52:07.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.854+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c0001010 con 0x7f98d00a98f0 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.855+0000 7f98b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.855+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2161350655 0 0) 0x7f98c0004bf0 con 0x7f98d00a98f0 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.855+0000 7f98b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.855+0000 7f98b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.855+0000 7f98b7fff640 10 monclient: dump: 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:fsid acf7b4d8-7b11-4409-8ed5-296e15695260 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-25T19:39:44.353122+0000 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 3 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T19:52:07.855 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-25T19:52:07.856 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-25T19:52:07.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.855+0000 7f98cd7fa640 4 set_mon_vals no callback set 2024-03-25T19:52:07.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.856+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2326301395 0 0) 0x7f98c001abc0 con 0x7f98d00a98f0 2024-03-25T19:52:07.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.856+0000 7f98ce7fc640 10 monclient: get_auth_request con 0x7f98bc06de90 auth_method 0 2024-03-25T19:52:07.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.856+0000 7f98b7fff640 10 client.4749.objecter ms_handle_connect 0x7f98bc06de90 2024-03-25T19:52:07.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.861+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1119940353 0 0) 0x7f98c0054940 con 0x7f98d00a98f0 2024-03-25T19:52:07.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.861+0000 7f98b7fff640 10 client.4749.objecter ms_dispatch 0x7f98d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:07.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.861+0000 7f98b7fff640 3 client.4749.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:07.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.861+0000 7f98b7fff640 3 client.4749.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:07.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.861+0000 7f98b7fff640 20 client.4749.objecter dump_active .. 0 homeless 2024-03-25T19:52:07.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.861+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98c0050280 con 0x7f98d00a98f0 2024-03-25T19:52:07.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.861+0000 7f98b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:07.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:07.861+0000 7f98b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:08.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.101+0000 7f98d596c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:08.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.101+0000 7f98d596c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:08.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.101+0000 7f98d596c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-25T19:52:08.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.101+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98d013a040 con 0x7f98d00a98f0 2024-03-25T19:52:08.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.120+0000 7f98b7fff640 1 -- 172.21.15.71:0/396576583 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2363388072) 0x7f98d013a040 con 0x7f98d00a98f0 2024-03-25T19:52:08.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.121+0000 7f98b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:08.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.121+0000 7f98b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:08.121 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-25T19:52:08.121 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"acf7b4d8-7b11-4409-8ed5-296e15695260","created":"2024-03-25T19:40:38.243750+0000","modified":"2024-03-25T19:50:43.107849+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:43.107849+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d573c566-53b7-4309-b868-cd9103387d76","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8cc2e952-5476-4395-9470-bfb55e5fe490","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"148f13c2-1acc-4900-9eaf-a56547ce9bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3dd90f4d-7c4d-4e88-9b19-9c89bd1b0375","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"504df1e2-17c5-4070-bcf7-ecce4030c19f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9a964fdb-6a47-40ef-92ac-d6ebe592e84d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d50af945-a618-4993-97bc-b141ea9f3237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f787a624-5765-4ba1-8afd-d5497f011748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.123+0000 7f98d596c640 10 librados: watch_flush enter 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.123+0000 7f98d596c640 10 librados: watch_flush exit 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.123+0000 7f98d596c640 20 client.4749.objecter shutdown clearing up homeless session... 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.123+0000 7f98d596c640 10 client.4749.objecter successfully canceled tick 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.123+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 >> v1:172.21.15.71:6800/3911735624 conn(0x7f98bc06de90 legacy=0x7f98bc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.123+0000 7f98d596c640 10 monclient: shutdown 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.123+0000 7f98d596c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.123+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 >> v1:172.21.15.71:6789/0 conn(0x7f98d00a98f0 legacy=0x7f98d0135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.124+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 shutdown_connections 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.124+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 >> 172.21.15.71:0/396576583 conn(0x7f98d00b7ea0 msgr2=0x7f98d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:08.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.124+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 shutdown_connections 2024-03-25T19:52:08.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.124+0000 7f98d596c640 1 -- 172.21.15.71:0/396576583 wait complete. 2024-03-25T19:52:08.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-25T19:52:08.124+0000 7f98d596c640 1 librados: shutdown 2024-03-25T19:52:08.135 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:08.135 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:52:08.136 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:52:08.136 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:52:08.137 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:08.498 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:51.254 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.511 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:51.267 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.558 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:11:51.315 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.566 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:51.286 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.571 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:51.327 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.574 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:51.293 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.582 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:51.338 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.610 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:51.329 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.617 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:11:51.374 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:09.996 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:11:25.150 35651== 2024-03-25T19:52:09.997 INFO:tasks.ceph.osd.0.smithi071.stderr:==00:00:11:25.150 35651== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:10.036 INFO:tasks.ceph.osd.0.smithi071.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:14.238 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:52:14.239 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:52:14.239 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:52:14.239 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:16.125 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:11:31.290 35644== 2024-03-25T19:52:16.125 INFO:tasks.ceph.osd.1.smithi071.stderr:==00:00:11:31.290 35644== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:16.163 INFO:tasks.ceph.osd.1.smithi071.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:20.341 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:52:20.341 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:52:20.342 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:52:20.342 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:22.208 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:11:37.357 35656== 2024-03-25T19:52:22.208 INFO:tasks.ceph.osd.2.smithi071.stderr:==00:00:11:37.357 35656== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:22.247 INFO:tasks.ceph.osd.2.smithi071.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:26.443 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:52:26.443 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:52:26.444 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:52:26.444 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:28.227 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:11:43.373 35663== 2024-03-25T19:52:28.228 INFO:tasks.ceph.osd.3.smithi071.stderr:==00:00:11:43.373 35663== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:28.264 INFO:tasks.ceph.osd.3.smithi071.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:28.527 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:12:11.283 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.547 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:12:11.304 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.574 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:12:11.331 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.577 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:12:11.333 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.578 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:12:11.334 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.600 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:12:11.357 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.605 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:11.324 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.621 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:11.340 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.650 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:11.369 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:32.545 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:32.545 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:52:32.546 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:52:32.546 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:52:32.546 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:34.648 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:11:49.811 35606== 2024-03-25T19:52:34.648 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:11:49.811 35606== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:34.686 INFO:tasks.ceph.osd.4.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:38.648 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:52:38.648 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:52:38.649 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:52:38.649 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:40.655 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:11:55.806 35609== 2024-03-25T19:52:40.656 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:11:55.806 35609== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:40.692 INFO:tasks.ceph.osd.5.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:44.750 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:44.751 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:52:44.751 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:52:44.751 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:52:44.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:46.470 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:12:01.631 35608== 2024-03-25T19:52:46.470 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:12:01.631 35608== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:46.509 INFO:tasks.ceph.osd.6.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:48.532 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:12:31.289 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.576 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:31.295 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.576 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:12:31.333 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.579 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:31.298 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.594 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:31.313 35062== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.597 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:12:31.353 35315== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.622 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:12:31.378 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.625 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:12:31.382 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.642 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:12:31.398 35319== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:50.853 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:52:50.853 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:52:50.854 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:52:50.854 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:52.409 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:12:07.570 35607== 2024-03-25T19:52:52.409 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:12:07.570 35607== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:52.440 INFO:tasks.ceph.osd.7.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:56.955 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:52:56.956 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:52:56.956 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:52:56.956 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:52:56.956 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:56.957 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-25T19:52:56.958+0000 7efe28e04640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35324) UID: 0 2024-03-25T19:52:56.981 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:52:56.982 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:52:56.982 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:57.009 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:52:57.009 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:52:57.009 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:52:57.009 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:57.036 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-03-25T19:52:57.026+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35284) UID: 0 2024-03-25T19:52:57.036 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-03-25T19:52:57.027+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:55:39.680 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:15:22.436 35315== 2024-03-25T19:55:39.681 INFO:tasks.ceph.mon.a.smithi071.stderr:==00:00:15:22.436 35315== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:55:39.689 INFO:tasks.ceph.mon.a.smithi071.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:55:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:55:41.748 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:55:41.748 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:55:41.749 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:55:41.749 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:55:41.766 INFO:tasks.ceph.mon.c.smithi071.stderr:2024-03-25T19:55:41.752+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35286) UID: 0 2024-03-25T19:55:41.766 INFO:tasks.ceph.mon.c.smithi071.stderr:2024-03-25T19:55:41.754+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:55:42.394 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:15:25.150 35319== 2024-03-25T19:55:42.394 INFO:tasks.ceph.mon.c.smithi071.stderr:==00:00:15:25.150 35319== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:55:42.403 INFO:tasks.ceph.mon.c.smithi071.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:55:47.850 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:55:47.850 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:55:47.851 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:55:47.851 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:55:47.851 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:55:47.868 INFO:tasks.ceph.mon.b.smithi193.stderr:2024-03-25T19:55:47.857+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35058) UID: 0 2024-03-25T19:55:47.868 INFO:tasks.ceph.mon.b.smithi193.stderr:2024-03-25T19:55:47.858+0000 9f86640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:55:48.528 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:15:31.247 35062== 2024-03-25T19:55:48.528 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:15:31.247 35062== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:55:48.538 INFO:tasks.ceph.mon.b.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:55:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:55:53.953 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:55:53.954 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:55:53.954 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:55:53.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:55:54.065 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:55:54.065 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:55:54.125 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:55:54.125 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:55:54.174 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:55:54.175 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:55:54.223 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:55:54.223 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:55:54.301 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:55:54.301 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:55:54.425 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:55:54.425 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:55:54.513 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:55:54.513 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:55:54.570 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:55:54.570 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:55:54.629 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:55:54.630 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/data/mon.a.tgz 2024-03-25T19:55:54.630 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-25T19:55:54.646 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.MiCdeL5luT 2024-03-25T19:55:54.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MiCdeL5luT 2024-03-25T19:55:54.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.MiCdeL5luT 2024-03-25T19:55:54.835 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.MiCdeL5luT is 133KB 2024-03-25T19:55:54.887 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.MiCdeL5luT 2024-03-25T19:55:54.901 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/data/mon.c.tgz 2024-03-25T19:55:54.901 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-25T19:55:54.955 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.GdfqFNSQjy 2024-03-25T19:55:54.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.GdfqFNSQjy 2024-03-25T19:55:55.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.GdfqFNSQjy 2024-03-25T19:55:55.132 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.GdfqFNSQjy is 164KB 2024-03-25T19:55:55.184 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.GdfqFNSQjy 2024-03-25T19:55:55.198 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/data/mon.b.tgz 2024-03-25T19:55:55.199 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-03-25T19:55:55.216 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.xkaEQO5NgG 2024-03-25T19:55:55.217 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.xkaEQO5NgG 2024-03-25T19:55:55.322 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.xkaEQO5NgG 2024-03-25T19:55:55.411 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.xkaEQO5NgG is 154KB 2024-03-25T19:55:55.463 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.xkaEQO5NgG 2024-03-25T19:55:55.479 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:55:55.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:55:55.481 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:55:55.872 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:55:56.062 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:55:56.064 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/remote/smithi071/crash 2024-03-25T19:55:56.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:55:56.093 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/remote/smithi193/crash 2024-03-25T19:55:56.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:55:56.120 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:55:56.120 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:55:56.137 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:55:56.161 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:55:56.161 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41965.log 2024-03-25T19:55:56.161 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2024-03-25T19:55:56.162 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-03-25T19:55:56.210 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41494.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41965.log.gz 2024-03-25T19:55:56.210 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41887.log 2024-03-25T19:55:56.210 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40831.log 2024-03-25T19:55:56.210 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39345.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz 2024-03-25T19:55:56.210 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41182.log 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41887.log.gz 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40831.log.gz 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41182.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41182.log.gz 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40168.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:55:56.211 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41104.log 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40870.log 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42357.log 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35642.log 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41104.log.gz 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2024-03-25T19:55:56.212 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40870.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.35642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39265.log 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42357.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39306.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.35642.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41807.log 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39265.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39659.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29472.log 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41650.log 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.tmp-client.admin.29472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29472.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41807.log.gz 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2024-03-25T19:55:56.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35893.log 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41650.log.gz 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39226.log 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41690.log 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.35893.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42201.log 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35893.log.gz 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39226.log.gz 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39501.log 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41690.log.gz 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41455.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42201.log.gz 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42123.log 2024-03-25T19:55:56.214 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39501.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42475.log 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40012.log 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39501.log.gz 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42475.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42123.log.gz 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39540.log 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40012.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42475.log.gz 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40012.log.gz 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40051.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42084.log 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39540.log.gz 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-03-25T19:55:56.215 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40363.log: /var/log/ceph/ceph-client.admin.42084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42162.log 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42084.log.gz 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42162.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42043.log 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42162.log.gz 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40714.log 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42043.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40714.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42043.log.gz 2024-03-25T19:55:56.216 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40714.log.gz 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40480.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39384.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40480.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41926.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41221.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39776.log 2024-03-25T19:55:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40675.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41926.log.gz 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42435.log 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42514.log 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41299.log 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39973.log 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42435.log.gz 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42514.log.gz 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39973.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:55:56.218 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41299.log.gz 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40441.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39973.log.gz 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40753.log 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.3.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41846.log 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41260.log 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35734.log 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41846.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41729.log 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41846.log.gz 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.35734.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41260.log.gz 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37899.log 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41729.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35734.log.gz 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40792.log 2024-03-25T19:55:56.219 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.37899.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41729.log.gz 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41611.log 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37899.log.gz 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42279.log 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40792.log.gz 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.41611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42396.log 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41611.log.gz 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42396.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42279.log.gz 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40285.log: gzip -5 --verbose 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42396.log.gz 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.40090.log 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40597.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35326.log 2024-03-25T19:55:56.220 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.40090.log: -5 --verbose 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.41572.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.35326.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41026.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40090.log.gz 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41572.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35326.log.gz 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41026.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39579.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.0.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41026.log.gz 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40948.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5/var/log/ceph/ceph-client.admin.39579.log: --verbose -- /var/log/ceph/ceph-client.admin.41533.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40948.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39579.log.gz 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.42240.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40948.log.gz 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41533.log: /var/log/ceph/ceph-client.admin.42240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log 2024-03-25T19:55:56.221 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41533.log.gz 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41416.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42240.log.gz 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.log: gzip 86.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gz --verbose 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39817.log 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41377.log: /var/log/ceph/ceph-osd.2.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.41338.log 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41338.log: /var/log/ceph/ceph-client.admin.39817.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39817.log.gz 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41338.log.gz 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42004.log 2024-03-25T19:55:56.222 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36780.log 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40129.log: /var/log/ceph/ceph-client.admin.42004.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42004.log.gz 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39031.log 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40519.log: /var/log/ceph/ceph-client.admin.36780.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36780.log.gz 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39618.log 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39031.log: /var/log/ceph/ceph-client.admin.39737.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39031.log.gz 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39856.log 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42318.log 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39856.log: /var/log/ceph/ceph-client.admin.39618.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39618.log.gz 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39856.log.gz 2024-03-25T19:55:56.223 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2024-03-25T19:55:56.224 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:55:56.224 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.42318.log: /var/log/ceph/ceph-client.admin.41768.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42318.log.gz 2024-03-25T19:55:56.224 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2024-03-25T19:55:56.224 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:55:56.224 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35296.log 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35296.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35296.log.gz 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/valgrind/osd.5.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log/var/log/ceph/valgrind/osd.6.log: 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/valgrind/mon.b.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35416.log 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/valgrind/osd.4.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35336.log 2024-03-25T19:55:56.225 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35216.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35416.log.gz 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35336.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph.audit.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.35336.log.gz 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35467.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35376.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35256.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35376.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35467.log.gz 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35256.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:55:56.226 INFO:teuthology.orchestra.run.smithi193.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35376.log.gz 2024-03-25T19:55:56.227 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35256.log.gz 2024-03-25T19:55:56.227 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-mgr.x.log: -- /var/log/ceph/ceph-client.admin.35507.log 2024-03-25T19:55:56.227 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:55:56.227 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35507.log: /var/log/ceph/ceph-mon.b.log: 86.7% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35507.log.gz 2024-03-25T19:55:56.227 INFO:teuthology.orchestra.run.smithi193.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:55:56.227 INFO:teuthology.orchestra.run.smithi193.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:55:56.259 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.c.log: 92.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:55:57.001 INFO:teuthology.orchestra.run.smithi071.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:55:57.075 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:55:57.082 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:55:57.122 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:55:57.319 INFO:teuthology.orchestra.run.smithi071.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:55:57.402 INFO:teuthology.orchestra.run.smithi071.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:55:57.411 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:55:57.423 INFO:teuthology.orchestra.run.smithi071.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:56:01.308 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:56:01.318 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:56:01.320 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-25T19:56:01.320 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m5.171s 2024-03-25T19:56:01.320 INFO:teuthology.orchestra.run.smithi071.stderr:user 0m14.326s 2024-03-25T19:56:01.320 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m0.538s 2024-03-25T19:56:01.436 INFO:teuthology.orchestra.run.smithi193.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:56:01.438 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T19:56:01.438 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m5.256s 2024-03-25T19:56:01.438 INFO:teuthology.orchestra.run.smithi193.stderr:user 0m8.835s 2024-03-25T19:56:01.438 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m0.342s 2024-03-25T19:56:01.438 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:56:01.439 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/remote/smithi071/log 2024-03-25T19:56:01.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:56:03.943 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/remote/smithi193/log 2024-03-25T19:56:03.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:56:05.491 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:56:05.702 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c676fccd97dc4dcf8c8523ad31fad595 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:56:05.707 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:56:05.716 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:56:05.731 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:56:05.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:56:05.746 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:56:05.832 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:56:05.841 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:56:05.841 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:56:05.844 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:56:05.858 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:56:05.861 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:56:05.863 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:56:05.863 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-25T19:56:05.863 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 41 -574us[ -574us] +/- 81ms 2024-03-25T19:56:05.864 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -189us[ -184us] +/- 67ms 2024-03-25T19:56:05.864 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +732us[ +732us] +/- 70ms 2024-03-25T19:56:05.864 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:56:05.866 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:56:05.866 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-25T19:56:05.866 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 -574us[ -574us] +/- 81ms 2024-03-25T19:56:05.866 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 7 377 106 -209us[ -293us] +/- 66ms 2024-03-25T19:56:05.866 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +720us[ +720us] +/- 70ms 2024-03-25T19:56:05.866 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:56:05.867 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:56:05.877 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:56:05.878 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:56:05.886 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:56:05.979 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:56:06.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:56:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:56:06.082 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:56:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:56:06.114 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:56:06.123 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:56:06.131 INFO:teuthology.task.internal:Duration was 5417.948725 seconds 2024-03-25T19:56:06.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:56:06.143 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:56:06.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:56:06.146 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:56:06.190 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:56:06.198 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:56:06.524 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:56:06.525 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-03-25T19:56:06.525 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:56:06.546 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T19:56:06.546 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:56:06.572 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:56:06.573 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:56:06.589 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:56:06.641 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:56:06.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:56:06.660 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:56:06.784 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:56:06.794 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:56:06.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:56:06.821 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:56:06.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:56:06.859 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:56:06.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:56:06.886 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-03-25T19:56:06.914 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-03-25T19:56:06.960 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:56:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:56:06.974 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:56:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:56:07.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:56:07.025 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:56:07.025 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/remote/smithi071 2024-03-25T19:56:07.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:56:07.064 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326/remote/smithi193 2024-03-25T19:56:07.064 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:56:07.099 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:56:07.099 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:56:07.102 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:56:07.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:56:07.162 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:56:07.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:56:07.171 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:56:07.172 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:56:07.174 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:56:07.186 INFO:teuthology.orchestra.run.smithi071.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:56 /home/ubuntu/cephtest 2024-03-25T19:56:07.204 INFO:teuthology.orchestra.run.smithi193.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:56 /home/ubuntu/cephtest 2024-03-25T19:56:07.205 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:56:07.215 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:56:07.234 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:56:07.259 DEBUG:teuthology.nuke:shortname: smithi071 2024-03-25T19:56:07.259 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:56:07.282 DEBUG:teuthology.nuke:shortname: smithi193 2024-03-25T19:56:07.283 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:56:07.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:12.047698', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQCzV6lpXKIBd4N/cJBTgzahLvMCwQAhXWL0LgcI3DO1snJHDmoyF4ex80DRwIxvCTQ+hZ070f+4Ry0HY0P/PM='} 2024-03-25T19:56:07.308 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621326', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:12.049502', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCknM59JyyEIUSXB1BqSUvecxOxnQEhCaT3/GXMIpAOQkfek8WYgZoMpKoTlmZYX5bmosbyObRrIa5xHthxUpxI='} 2024-03-25T19:56:07.338 INFO:teuthology.orchestra.console.smithi071:Power off 2024-03-25T19:56:07.339 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:56:07.363 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2024-03-25T19:56:07.363 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:56:07.380 INFO:teuthology.orchestra.console.smithi193:Power off 2024-03-25T19:56:07.380 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:56:07.391 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-25T19:56:07.404 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-03-25T19:56:07.404 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:56:07.428 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-25T19:56:11.392 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:56:11.429 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:56:11.502 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-25T19:56:11.540 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-25T19:56:15.506 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:56:15.542 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:56:15.617 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2024-03-25T19:56:15.652 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-25T19:56:15.718 INFO:teuthology.orchestra.console.smithi071:Power off completed 2024-03-25T19:56:19.654 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:56:19.764 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-03-25T19:56:19.865 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-03-25T19:56:19.970 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 5417.94872546196 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c676fccd97dc4dcf8c8523ad31fad595 status: fail success: false 2024-03-25T19:56:19.970 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:56:20.049 INFO:teuthology.run:FAIL